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(g> Method and apparatus for providing enhanced graphics in a virtual world. 

(g) Method and sytems are provided for render- 
ing and displaying in a real time 3-D computer 
graphic system a sequence of images of a 
subject using a plurality of time-sequenced text- 
ures such that at least a portion of the subject 
appears animated. The time-sequenced text- 
ures are derived from sources such as digitized 
frames or fields captured from a video record- 
ing of a live actor who may be engaged in a 
scripted performance, or a digitally-recorded 
cartoon animation sequence, and can be map- 
ped in different ways to different types of sur- 
face geometries to achieve animation. 
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BACKGROUND OF THE INVENTION 

This invention relates to a method and system for 
providing enhanced graphics in an interactive virtual 
world environment 

"Virtual reality" is a phrase that has been used to 
describe the experience of living or playing in a com- 
puter-generated environment which models a three- 
dimensional f3-D w ) virtual space (i.e., a virtual 
world). In virtual reality systems, a viewer typically 
dons a set of goggle- mounted video screens or some 
other form of head-mounted display ("HMD"), and an 
audio headset, to block out the real world. Typically, 
the HMD is a conventional pixel (picture element) 
based, raster scan video display device. The viewer 
might also be provided with a simulated laser gun, a 
vehicle with accelerator and brake pedals, or some 
other device coupled to the computer system to en- 
able the viewer to move about in, interact with or 
otherwise influence objects and characters in the vir- 
tual world. Sounds emanating from the virtual world 
(such as lasers firing or vehicles accelerating) are 
heard in the viewer's headset. 

On the HMD the viewer sees images of virtual 
world scenes that are generated from a 3-D model of 
the virtual world by a computer graphic system. The 
scenes are displayed to the viewer as they appear 
from a specific position and orientation in the virtual 
world, called the "viewpoint" or "eyepoint." Usually, 
the viewer is given some degree of control over the 
position and orientation of the viewpoint, thus allow- 
ing the viewer to see different images from a plurality 
of different viewpoints. 

By enabling the viewer to change the position 
and/or orientation of his or her viewpoint, the comput- 
er graphic system can create the illusion of the viewer 
"travelling" through the virtual world and looking in all 
directions. Depending on the capabilities and pro- 
gramming of the system, the viewer might be allowed 
to "travel" without restriction above, below and around 
a scene, as well as into or through structures, as 
though the viewer could fly or had other capabilities. 
The system might also be designed to constrain the 
motion of the viewpoint in various ways to achieve re- 
alism or to further the plot of a story being enacted 
around the viewer in the virtual world. For example, 
t he viewer might only be allowed to position and orient 
the viewpoint no closer than six feet from the ground 
to simulate a view of the virtual world from the van- 
tage of a person standing on the ground in the world. 
Alternatively or in addition, the viewpoint might be 
constrained from passing through the image of a solid 
surface (such as the wall of a building) - just as it is 
typically impossible in the real world to walk through 
the wall of a building. Also, the viewer might be con- 
strained to move along a defined path, as though 
traveling on a train, such that a series of events can 
be enacted as the viewer's train passes by predeter- 



mined positions in the virtual world. 

Virtual reality systems have developed from mili- 
tary and commercial airline flight simulators, and mili- 
tary tank simulators, in which computer graphic sys- 

5 terns render a simulated, 3-D world from the perspec- 
tive of a person looking out of an aircraft cockpit win- 
dow or tank turret (i.e., the system's "viewpoint"). The 
world created by such simulators typically includes 
static structures and terrain (e.g., an airport with run- 

10 ways and buildings situated in a world including lakes, 
rivers and mountains), and moving objects (e.g., fly- 
ing aircraft, land vehicles and clouds). The images of 
the simulated world displayed on the "windows" of 
the cockpit or turret continually change ~ as might oc- 

15 cur in the real world ~ in response to changes in pos- 
ition and attitude of the aircraft or vehicle being 
"flown" or "driven" by the person participating in the 
simulation. 

Virtual reality systems have applicability in the 
20 entertainment industry. Computer games and arcade 
machines presenting a virtual reality experience as a 
form of entertainment can provide a viewer with a 
high level of enjoyment Virtual reality systems can 
immerse a viewer in a realistic world, or a highly fan- 
25 tasized or magical one where even the laws of phys- 
ics have been skewed, to produce an entertainment 
experience available by no other means. 

In conventional virtual reality systems, an image 
of a virtual world scene displayed to a viewer is typi- 
30 cally created as follows. First, a library (or database) 
containing 3-D data representative of objects in the 
virtual world is created and stored, to be used in gen- 
erating all images of the virtual world. Then, when an 
image of a virtual world scene is to be displayed from 
35 a particular viewpoint, the data in the library is proc- 
essed to render objects within the viewer's field of 
view as a perspective image on a two-dimensional 
viewplane as they would appear to a viewer from that 
viewpoint. As used herein, "perspective image" 
40 means an image that is drawn such that volumes and 
spatial relationships in the image are depicted in ac- 
cordance with what a person would see from the 
viewpoint if the virtual world scene were actually re- 
produced in 3-D space instead of on a 2-D viewplane 
45 (e.g., the size of an image of an object in the virtual 
world changes in inverse proportion to changes in 
distance between the object and the viewpoint). 

One conventional technique used in virtual reality 
systems (and other computer graphic systems as 
so well) for creating a library of 3-D objects is to model 
the objects by dividing each into one or more surfaces 
defined relative to a local coordinate system which 
can then be positioned relative to a fixed coordinate 
system (called the "world coordinate system"). These 
55 surfaces can be polygons or other geometric figures 
(e.g., spherical sections, bi-cubic patches) of various 
sizes and shapes configured to represent the shapes 
of objects in the virtual world. If a plurality of these 
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polygons (or other geometric figures) are combined, 
substantially any 3-D shape can be modeled. Com- 
plex surfaces, including curved or perforated surfac- 
es, can be piece-wise approximated using sufficient 
numbers of small polygons (or other geometric fig- 5 
ures). 

Realism and detail are added to a displayed im- 
age of a virtual world scene by texturing the modeled 
objects in the scene. Texturing gives the components 
of the displayed image patterned physical attributes. 10 
The surfaces of modeled objects are typically tex- 
tured by generating 2-D arrays of scalar values ("tex- 
ture maps") which represent features to be overlaid 
on the geometric figures that represent the visible 
surfaces. The combination of the geometric figures 15 
and overlaid texture map forms a composite which is 
displayed, pixel by pixel, on the viewer's video display 
device. Individual elements of a texture map may cor- 
respond on a one-to-one, many-to-one or one-to-ma- 
ny basis with pixels of the display device. Such a tex- 20 
ture element is conventionally referred to as a "texel." 
Texture maps may include information defining one or 
more surface attributes such as color (represented, 
for example, by red, green and blue data elements), 
local deviation of the normal of the surface of the 25 
polygon (i.e. "bump maps," which simulate bumpy 
surfaces through illumination cues), local displace- 
ment along the normal of the surface, and luminosity. 
Further, a single map may contain multiple represen- 
tations of the surface such as its appearance at drf- 30 
ferent ranges (e.g., "MIP" maps) or different angles 
(e.g., a map containing images of an object as viewed 
from different compass directions such as NNW and 
East). 

Several other techniques for modeling 3-D ob- 35 
jects in a computer database are known. These in- 
clude, for example, techniques based on representing 
the surfaces of the objects as mathematical functions 
or volume elements (conventionally called "voxels"). 
Such techniques tend to require more intensive arith- 40 
metical calculation than polygon-based modeling and 
are less commonly used today in conventional com- 
puter graphic systems. 

Conventional computer graphic systems use va- 
rious techniques for rendering a 3-D scene as per- as 
spective images on a two-dimensional viewplane 
from a 3-D database of modeled objects. Like the 
models themselves, these different techniques vary 
widely in the amount of computational complexity in- 
volved. A relatively simple technique, called "the pain- so 
ter's algorithm," is to render a pixel-based image of 
each object in the scene in its proper position, orien- 
tation and perspective according to the viewer's view- 
point, proceeding from the back of a scene to the front 
and "painting" the pixel data for objects in the fore- 55 
ground over that of previously drawn objects in the 
background. Another common technique, known as 
"z-buffering", involves writing for each pixel a "z" or 



range value which is used to resolve object priority. 
This technique is widely used in conventional comput- 
er graphic systems. Still another technique, known as 
"ray casting", is to render the scene image by project- 
ing a straight ray backwards from the eyepoint 
through each pixel of the display to the first surface 
struck by each ray. As each ray intersects a surface, 
it accumulates the properties of the surface and adds 
them to the corresponding pixel. This technique re- 
quires more arithmetic calculation and is less com- 
monly used than the painter's algorithm and z-buffer- 
ing techniques. Other rendering methods, such as 
ray tracing and radiosity, have been developed to add 
shadows, diffuse lighting effects and reflections. 
These methods are capable of producing very realis- 
tic images, but require very large amounts of arith- 
metic calculation to do so. 

The calculation required by many of these tech- 
niques may cause a conventional computer graphic 
system to take hours, if not days, to compute a single 
image. Accordingly, realism is usually severely con- 
strained in images generated by conventional virtual 
reality systems, which must compute the images in 
"real time," meaning that images must be computed 
in a fraction of a second (e.g., at a rate of 10-60 frames 
per second) so that the viewer, and other objects, can 
move smoothly through the virtual world in approxi- 
mately the same amount of time that they actually 
move in the real world. 

A particularly detrimental aspect of the limited 
graphic realism of prior virtual reality systems is the 
inability of such systems to generate realistically ani- 
mated human actors in the synthetic 3-D computer- 
generated environment. Especially important in this 
regard are fine details, such as facial expressions, 
and realistic movement of the images. In some 3-D 
computer graphic systems, articulated figures are 
generated using hierarchical 3-D models to display 
human actors. This type of animation, however, tends 
to appear "robot-like." Further, such animation is 
computationally expensive. 

A related drawback of prior real time 3-D comput- 
er graphic systems that use texture maps is that they 
require the texture map patterns, like object geometry 
data, to be pre -computed and stored in a database of 
the 3-D graphics subsystem. In such systems, texture 
map patterns typically are not generated dynamically 
(this does not include and is in contrast to convention- 
al techniques for rendering textures procedurally, i.e. 
techniques which compute a texture pixel-by-pixel at 
render time by evaluating some mathematical func- 
tion). As a result, texture map patterns typically can- 
not be altered by a viewer, or changed in response to 
a viewer's actions (i.e., the texture map patterns are 
not interactive). Yet, it might be desired that a 3-D vir- 
tual world be populated with objects that simulate real 
world objects having surfaces which ordinarily 
change appearance in response to a person's ac- 
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tions, such as a computer console having a screen 
that displays different text or images responsive to a 
viewer's inputs at a keyboard. 

In view of the foregoing, it would be desirable to 
be able to provide a computer graphic method and 
system which enhances a computer-generated, vir- 
tual reality environment with realistically animated hu- 
man actors or other live beings or dynamic objects. 

It also would be desirable to be able to provide a 
computer graphic method and system which enhanc- 
es a computer-generated, virtual reality environment 
with dynamically generated textures. 
The invention is defined by the independent claims to 
which reference should be made. 

Embodiments of the invention may enhance a 
computer-generated, virtual reality environment with 
realistically portrayed, animated photo-likenesses of 
real-world human actors or other live beings or dy- 
namic objects, or cartoon animation. 

Embodiemnts of the invention may apply dynam- 
ically generated textures to surfaces in a virtual real- 
ity environment. 

In a preferred embodiment of the invention, a se- 
quence of images of a surface with a plurality of time- 
sequenced textures is rendered and displayed in a 
real time 3-D computer graphic system such that at 
least a portion of the surface appears animated. Data 
representing a surface to be textured are stored in the 
computer graphic system. Also stored or generated 
dynamically are data representative of a plurality of 
time-sequenced textures (such as digitized frames or 
fields captured from a video recording of a live actor 
who may be engaged in a scripted performance, or a 
digitally-recorded cartoon animation sequence) to be 
applied to the surface. Based on the data represen- 
tative of the surface and the data representative of 
subsequent ones of the plurality of textures, a se- 
quence of images (such as a realistic, photo-likeness 
of a human being) then is displayed in real time on at 
least a portion of the surface, with the texture chang- 
ing so that the image appears animated. 

Animated textures can be displayed on planar 
and non-planar surfaces. Such surfaces may com- 
prise one or a plurality of primitive geometric elements 
(e.g., polygons, bi-cubic patches, etc.). An illustrative 
embodiment is described in which an animated tex- 
ture of a human face is applied to a virtual 3-D man- 
nequin's head. In an alternative preferred embodi- 
ment only a portion of the texture is animated, thereby 
allowing the eyes or mouth of the mannequin to be 
moved (and emphasized) by animation while the re- 
maining parts of the textured face remain still. In eith- 
er case, the geometry of the surface may change 
over time, such that a talking head may be trans- 
formed into a talking globe. As with a conventional 
static texture, the textures of the animation sequence 
stay aligned with the changing geometry. 

To further enhance the realism of the animated 



image in the virtual world, actors are filmed from a 
position in the real world, and they translationally 
move in the real world, in a scaled analog fashion to 
what their position and translation^ movements will 
5 be in the virtual world relative to the viewpoint More- 
over, animated images can be made viewer- interac- 
tive by switching between different sets of stored or 
dynamically generated time-sequenced textures. 

10 BRIEF DESCRIPTION OF THE DRAWINGS 

Acomplete understanding of the invention and its 
advantages may be gained from consideration of the 
following detailed description, taken in conjunction 
is with the accompanying drawings, in which like refer- 
ence characters refer to like parts throughout, and in 
which: 

FIG. 1 is a simplified block diagram of an exem- 
plary computer graphic system in which the 
20 graphics techniques of the present invention can 

be implemented; 

FIG. 2 shows how textures can be interchanged 
in the (u,v) coordinate texture space of the com- 
puter graphic system of FIG. 1 in accordance with 

25 the present invention; 

FIG. 3 shows an exemplary method by which a 
scripted performance of a live actor or other en- 
tity physically existing in the real world may be 
captured on a video recording to create animated 

30 textures for use with the present invention; and 

FIG. 4 is a block diagram illustrating a method for 
generating data representative of a time- 
sequenced plurality of textures from a video re- 
cording of an entity or event in the real world in 

35 accordance with principles of the present inven- 

tion. 

DETAILED DESCRIPTION OF THE INVENTION 

40 The present invention is described below in the 

context of an interactive 3-D computer graphic system 
embodiment that uses polygons and texel-based tex- 
ture maps to render images. The particular techni- 
ques by which a library of polygon data and texture 

45 maps are generated, and by which 2-D images are 
drawn using these data, are not discussed with par- 
ticularity in order to avoid unnecessarily encumbering 
the following discussion with a description of techni- 
ques that are conventional and well known to those 

so skilled in the art It will be readily understood by per- 
sons skilled in the art, however, that the present in- 
vention can be used with 3-D computer graphic sys- 
tems which use voxel, spline (i.e. mathematical func- 
tions) or other surface modeling techniques, and 

55 which use painter's algorithm, 2- buffering, ray cast- 
ing or other rendering techniques. 

A conventional interactive 3-D computer graphic 
system typically includes at least two major subsys- 
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terns: a graphics subsystem and a CPU subsystem. 
The graphics subsystem generates images and out- 
puts the images to a display device. The CPU subsys- 
tem receives input from viewer interaction devices 
and describes to the graphics system what is to be 
displayed. The present invention is preferably imple- 
mented in a system having this type of system archi- 
tecture. 

FIG. 1 shows a simplified block diagram of an ex- 
emplary embodiment 100 of an interactive 3-D com- 
puter graphic system in which the present invention 
can be implemented. In system 100, the graphics 
subsystem comprises rendering pipeline 102 (with 
associated geometry memory 124 and texture mem- 
ory 126) and rendering control process(or) block 104, 
and the CPU subsystem comprises scenario proc- 
essor) block 106 and I/O interface process(or) block 
108. 

Rendering pipeline 102 converts object data 
(e.g., polygon and texture data) into image data rep- 
resentative of the perspective view of virtual world 
scenes for particular viewpoints. The conversion 
process typically involves object geometry transfor- 
mations, viewing transformations, clipping, rasteriza- 
tion and other conventional graphics processing op- 
erations. Rendering pipeline 102 is preferably imple- 
mented using a multiprocessing architecture to ach- 
ieve high-speed rendering in accordance with con- 
ventional real time computer graphics techniques. 

In accordance with such conventional techni- 
ques, rendering pipeline 102 calculates the pixel data 
for an image by rendering surfaces representative of 
virtual- world objects within the viewer's field of view. 
The surfaces typically comprise three- or four- sided 
colored polygons, any of which is capable of being 
textured or smoothly shaded, and modeled as trans- 
parent or semi-transparent (translucent). A fully trans- 
parent surface would, when rendered, be invisible in 
the display, while a semi-transparent surface would 
visibly obscure (but not completely occlude) a back- 
ground. The polygon and texture data on which ren- 
dering pipeline 102 operates reside respectively in 
geometry memory 124 and texture memory 126. 

Polygon data are stored in geometry memory 1 24 
in a plurality of hierarchical data structures defining 
various object geometries. These object geometries 
may range in size from one polygon (representing, 
e.g.. a very simple virtual world entity, or object) to 
thousands of polygons (representing, e.g., a portion 
of a complex terrain model covering hundreds of 
square miles of scenery or an elaborate architectural 
model). 

Texture data are stored in texture memory 126 
usually in the form of rectangular texture maps. Each 
rectangular texture map in texture memory 126 pos- 
sesses its own two-dimensional coordinate space, 
conventionally referred to as a "u,v" space or texture 
space. These texture maps conventionally comprise 



digitized images that are mapped onto surfaces de- 
fined by the polygon data in geometry memory 124. 
Various algorithms are known for performing image 
transformations of this type. For example, a texture 

5 can be mapped to a polygon in geometry memory 124 
by associating with each vertex of the polygon a (u.v) 
coordinate of a particular texture map, and linearly in- 
terpolating (u,v) coordinate values for intermediate 
points across the polygon surface. Other, more so- 

10 phisticated "projection" algorithms have been devel- 
oped and are conventionally used to transform the 
(u,v) coordinates of texels in a texture map to coordin- 
ates on a 3-D object surface. As will be readily appa- 
rent, the graphics techniques of the present invention 

is can be used with substantially all such mapping algo- 
rithms. 

At the tail end of rendering pipeline 102, image 
data are converted to video signals which are provid- 
ed to display 122. Display 122 may comprise a head- 

20 mounted display device and/or a non-encumbering 
device such as a free-standing or wall-mounted mon- 
itor. In either case, the display device may be a direct- 
view or projection type device. The present invention 
also can be used with 3-D displays as well, such as 

25 multi-planar and stereo pair displays. For high quality 
entertainment, it is important that the viewer's display 
device have a suitably high number and density of pix- 
els, and that rendering pipeline 102 be capable of pro- 
viding imagery of like resolution. A display of 1000 

30 (horizontal) by 800 (vertical) pixels provides a prefer- 
able image although, of course, a greater or lesser 
pixel density may be used. 

Rendering pipeline 102 operates under the con- 
trol of rendering control process(or) block 104. More. 

35 particularly, during run time rendering control proc- 
essor) block 1 04 controls the loading of data into ge- 
ometry memory 124 and texture memory 126 and the 
timing of rendering cycles of rendering pipeline 102. 
Geometry memory 124 and texture memory 126, 

40 which preferably comprise high-speed electronic 
memories, ordinarily do not store the polygon and tex- 
ture data for an entire virtual world. Instead, the data 
loaded into these memories are usually selected from 
a larger run- time database stored on a secondary 

45 storage device and transferred into geometry mem- 
ory 124 and texture memory 126 as needed to render 
objects within the visible range of the viewer. This se- 
lection process is normally the responsibility of ren- 
dering control process(or) block 104, which selects 

so the data based on commands or procedure calls from 
scenario process(or) block 106 describing the scene 
to be rendered. Rendering control processor) block 
104 keeps track of the objects in the scene by main- 
taining a list of information for each object, such as its 

55 current position, orientation and status. The list also 
includes for each object a pointer 216 to a particular 
object geometry that is to be used to render the ob- 
ject This list is maintained in a memory 120 in render- 
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ing control process (or) block 1 04. 

In many conventional computer graphics sys- 
tems, the run-time database is stored on an external 
secondary storage device, as illustrated in FIG. 1 by 
run-time database 110. In this arrangement, render- s 
ing control process(or) block 104 is usually directly 
burdened with the task of moving data from the data- 
base to the electronic memories of geometry memory 
124 and texture memory 126 during run-time. Some, 
or all, of this burden can be relieved by providing ge- w 
ometry memory 124 and texture memory 126 with in- 
ternal secondary stage devices coupled to the elec- 
tronic memories by dedicated DMA channels. The in- 
ternal secondary storage devices (e.g., disk drives) 
may store part or all of the geometry and texture data is 
needed for a particular virtual world, and are operated 
by conventional DMA controllers under the control of 
rendering control process(or) block 104. In FIG. 1, 
such devices and their associated DMA controllers 
are shown as blocks 124a and 126a. 20 

As previously mentioned, scenario process(or) 
block 106 controls the image generated by the graph- 
ics subsystem by issuing commands or procedure 
calls to rendering control process(or) block 104 to de- 
scribe a particular scene. Some of these commands 25 
or procedure calls specify the attributes and place- 
ment of objects in the scene, while others specify the 
viewpoint and other aspects of the image. These 
commands or procedure calls are issued based on in- 
structions contained in a control program which see- 30 
nario process(or) block 106 executes. Viewpoint is 
determined by scenario process (or) block 106 from 
viewer interaction data provided by I/O interface proc- 
essor) block 108. 

The viewer interaction data provided by I/O inter- 35 
face process(or) block 108 may include data from one 
or more input devices. Such devices may include, for 
example, a viewer head sensor 116 for tracking the 
position and orientation of a viewer's head and viewer 
activity sensors 118 for receiving inputs from the 40 
viewer indicative of a desired direction and rate of 
travel through the virtual world. Conventional sens- 
ing devices of various well-known types can be used 
to implement these sensors. For instance a suitable 
head tracking device called "the Bird" is commercially 45 
available from Ascension Technology Corp.. of Col- 
chester, Vermont. 

As shown in FIG. 1, position and orientation data 
from viewer head sensor 116 alternatively may be 
coupled directly to scenario process(or) block 106 to so 
minimize data transport delay. This configuration is 
preferred because, as will be discussed below, it is 
important that scenario pro cess (or) block 106 have 
the most current information available as to the pos- 
ition/orientation of the viewer's head when directing 55 
the graphics subsystem. Thus, delay is avoided by 
bypassing I/O interface pro cess (or) block 108. 

The architecture of FIG. 1 is substantially typical 



of numerous conventional interactive 3-D computer 
graphic systems. In such systems, rendering pipeline 
102 and process(or) blocks 104, 106 and 108 are im- 
plemented in various ways using general purpose 
and/or specialized computer hardware. For example, 
separate computers, or separate processors within a 
multiprocessor computer, may be used for each 
block. Alternatively, two or more of the blocks may 
run as parts of the same process, or as different proc- 
esses on a common processor. The present invention 
can be used in substantially any of these implemen- 
tations. 

In a preferred embodiment, at least the graphics 
portion of system 100 (i.e. rendering pipeline 102 and 
rendering control process(or) block 104), and most 
preferably system 100 in its entirety, is implemented 
using a specialized computer hardware device of the 
type commonly referred to as an "image generator". 
Image generator devices are commercially available 
from many companies, including: Ball Systems Engin- 
eering Division, of San Diego, California; Bolt, Bra- 
nek, Newman of Redland, Washington; Division Ltd. 
of Bristol, England; Evans & Sutherland Computer 
Corporation ("E&S") of Salt Lake City, Utah; G.E Ae- 
rospace Simulation and Control Systems Department 
of Daytona Beach, Florida; and Silicon Graphics, Inc. 
of Mountain View, California. An image generator 
having two separate channels for producing video 
outputs is particularly preferred. Such hardware can 
be used to render two views of an image from slightfy 
different viewpoints (e.g., 3" apart) at approximately 
the same time to provide a stereoscopic HMD display. 
However, a single channel image generator is also 
sufficient for producing a high quality monoscopic 
display. Multiple channel devices also can be used to 
provide multiple viewers with different concurrent 
views of the same virtual world. Of course, sufficient- 
ly powerful general purpose computers could be used 
instead of a special purpose image generator without 
departing from the principles of the present invention. 

Communication among rendering pipeline 102 
and process(or) blocks 104, 1 06 and 108 must be ac- 
complished at high speed to properly effect a real 
time system. Substantially any conventional high- 
speed communication technique can be used, al- 
though the choice of technique largely depends on 
the particular manner in which the process(or) blocks 
are implemented. As one example, Ethernet can be 
used as in conventional computer graphics systems 
to communicate between different computers. A pre- 
ferred technique is to use shared memory (i.e., a 
memory space that is addressable by each proc- 
essor) block to which it is coupled as though the 
memory space were part of the main memory of the 
respective process(or) block). If the process(or) 
blocks of system 100 are implemented using different 
computer devices, substantially any commercially 
available dual port shared memory card which suits 
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the purpose of the memory (such as bus adapter 
products of Bit3 Computer Corporation of Minneapo- 
lis, Minnesota) can be used to communicate between 
the computers. 

A shared memory space can be divided into mul- 
tiple portions having different configurations de- 
signed to serve different purposes. For example, cir- 
cular buffers or other queuing configurations can be 
used to send control and status messages between 
process(or) blocks. This permits asynchronous com- 
munication between process(or) blocks and prevents 
loss of an important command or message if one 
process(or) block momentarily becomes too busy to 
keep up with communications from the other. 

In addition, individually addressable memory 
mapped "mailboxes" can be used for communicating 
data that are repeatedly updated. For example, each 
time I/O interface process(or) block 108 receives pos- 
ition data from viewer activity sensors 118, the data 
can be stored by I/O interface process(or) block 108 
into a designated mailbox in a memory shared by sce- 
nario process(or) block 106 and I/O interface proces- 
sor block 108. When updating such position data, I/O 
interface process(or) block 1 08 overwrites whatever 
data was previously stored in the mailbox. Ordinarily, 
scenario process(or) block 106 reads the data in the 
mailboxes between updates. However, rf scenario 
processor) block 1 06 becomes too busy to do so, and 
thus misses a particular update, the overall average 
rate at which the data are updated and read is suffi- 
ciently high that the missed update is not significant. 

As another example, memory 120 also can be im- 
plemented as a shared memory, such that scenario 
process(or) block 106 has direct access to an object 
list (maintained in memory 120) used by rendering 
control process(or) block 104 to keep track of objects. 
This implementation allows scenario process(or) 
block 106 to modify the scene to be rendered by the 
graphics subsystem by directly changing data in the 
object list. 

FIG. 2 shows an exemplary implementation of 
the present invention in system 100 of FIG. 1. In FIG. 
2, geometry memory 124 is shown as containing a 
plurality of object geometry data structures 202. 204, 
and 206. The object geometries include data defining 
the surfaces, in this case polygons, comprising the 
respective objects. A typical polygon definition in- 
cludes data defining the vertices of the polygon, as 
shown in FIG. 2 by the expanded polygon definition 
208. Each vertex definition of a polygon to be textured 
includes a coordinate (x.y.z) in object space and a 
corresponding coordinate (u,v) in texture space (i.e. 
in the texture map to which the polygon is associat- 
ed). A vertex definition may also include properties 
such as color and luminosity. Further properties that 
can be defined for a polygon are the type of texture 
to be associated with the polygon (e.g., full color), 'rf 
any, and the manner in which the texture is to be map- 



ped to the polygon (e.g., the mapping algorithm to be 
used). 

In accordance with the present invention, realis- 
tic animation is provided in a computer-generated 3- 
5 D world by interchanging a time- sequenced set of tex- 
tures on a rendered surface. One way in which this 
can be accomplished in system 100 is by associating 
a polygon to be textured with a particular memory lo- 
cation address within texture memory 126 which is 
10 used as a common reference point for the (u,v) coor- 
dinate systems of a plurality of time-sequenced tex- 
ture maps sequentially transferred into texture mem- 
ory 126. The time-sequenced texture maps can be 
transferred from a run-time database stored in an ex- 
15 ternal secondary storage device (e.g., run-time data- 
base 110) or a texture database stored in an internal 
secondary storage device such as storage unit 126a. 
The memory address used as a common reference 
point may correspond, for example, to t he memory lo- 
20 cation at which the origin of the (u,v) coordinate sys- 
tem for each of the maps is to be located. At arbitrary 
or periodic times between rendering cycles of the ren- 
dering pipeline 102, different ones of the plurality of 
texture maps are written into the same (u,v) coordin- 
25 ate space within texture memory 126 using the com- 
mon reference point to position and orient the (u,v) co- 
ordinate system of each map. When applying texture 
to the polygon, rendering pipeline 102 uses whatever 
texture data is in the (u,v) coordinate space associat- 
30 ed with the polygon, and maps that texture data to the 
polygon (e.g., in accordance with the (u,v) coordin- 
ates assigned to the vertices of the polygon) by the 
polygon's definition in the object geometry. Different 
sized maps can be included in the time-sequenced 
35 set of textures by normalizing the (u.v) coordinate 
systems of the texture maps. 

Various methods can be used to associate a poly- 
gon to be textured with a particular area in texture 
memory for purposes of associating the polygon to a 
40 particular (u,v) coordinate space. For example, as 
part of the object geometry information which is gen- 
erated to define the polygons forming an object, each 
polygon to be textured may include a pointer 210 that 
directly addresses a memory location in texture mem- 
45 ory 126 (as shown by polygon definition 208 of FIG. 
2). Alternatively, the pointer may address a particular 
area in texture memory 126 indirectly by pointing to a 
map number in a memory-resident look-up table 
which correlates the map number with a memory lo- 
50 cation in texture memory. 

In the system of FIG. 1, the process of inter- 
changing texture maps during run time preferably is 
managed by rendering control process(or) block 104 
under the direction of commands or procedure calls 
55 from scenario process(or) block 106. The specific im- 
plementation of a texture memory management sys- 
tem in accordance with principles of the present in- 
vention is a designer's choice and may vary from one 
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computer graphic system to another depending on 
the architecture of the particular system. For exam- 
ple, rendering control process(or) block 104 may be 
primarily responsible for much of the "bookkeeping" 
involved in transferring texture maps from run-time s 
database 110 or storage unit 126a into texture mem- 
ory 1 26, in which case scenario process(or) block 1 06 
may need only to tell rendering control process(or) 
block 104 the name of a particular texture that is to 
be applied to a particular surface. Rendering control 10 
process(or) block 104 would, in response to such a 
command or procedure call, correlate the texture 
name to a particular texture file in run-time database 
110 or storage unit 126a using a memory-resident 
cross-reference table, and then transfer the texture 15 
map in that file to the location in texture memory 126 
identified (either directly or indirectly) by a pointer as- 
sociated with the geometry for the surface specified 
in the command or procedure call (e.g., a pointer in- 
cluded in the object geometry stored in geometry 20 
memory 124). 

Alternatively, the bulk of such bookkeeping over- 
head involved in managing the texture memory can 
be transferred from rendering control process(or) 
block 1 04 to scenario process(or) block 1 06. This can 25 
be accomplished, for example, by providing scenario 
process(or) block 106 with file references and file lo- 
cation data for texture files in run-time database 110 
or storage unit 126a (e.g., during initialization of the 
computer graphic system). Scenario process(or) 30 
block 106 may then issue commands to rendering 
control process(or) block 104 during run time which 
instruct rendering control process(or) block 104 to 
transfer a specific texture map from a specific file lo- 
cation in run-time database 110 or storage unit 126a 35 
to a specific area in texture memory 126, using the 
identifying information previously provided during ini- 
tialization. 

This transfer to scenario process(or) block 106 of 
control over texture memory management may be 40 
necessary, or at least preferable, in some implemen- 
tations because of limitations in the processing capa- 
bility of the rendering control process(or) block of a 
conventional graphics subsystem. For example, ren- 
dering control process(or) block 104 may simply be 45 
too busy managing object geometries to perform the 
bookkeeping operations involved in managing the 
transfer of textures between run-time database 110 or 
storage unit 126a and texture memory 126. In such 
implementations the software of rendering control so 
process(or) block 1 04 can be modified to transfer tex- 
ture memory management to scenario process(or) 
block 106, which may have more available data proc- 
essing capacity. Such modifications are within the 
skill of persons of ordinary skill in the art conversant 55 
with the software of the particular graphics subsys- 
tem used. 

In implementations where scenario process(or) 



block 1 06 is given ultimate control over management 
of texture memory 126, the control program executed 
by scenario process(or) block 106 will include instruc- 
tions telling scenario process(or) block 106 when to 
issue commands to rendering control process(or) 104 
to load texture data into texture memory 126. In such 
an implementation, it is therefore the responsibility of 
the programmer to ensure that the appropriate tex- 
tures are loaded into texture memory 126 for a par- 
ticular virtual world scene. 

The texture maps that are transferred into texture 
memory 126 in accordance with principles of the 
present invention can, but need not, be pre-computed 
and compiled as part of a stored run-time database. 
In accordance with another aspect of the present in- 
vention, such textures may be transferred into texture 
memory 126 from a source other than a stored run- 
time database, thus permitting textures to be gener- 
ated dynamically (i.e. apart from and before or after 
the compilation of the run-time database). 

For example, texture data may be provided by 
scenario process(or) block 106 as part of a command 
or procedure call to rendering control process(or) 
block 104. These data may be stored in texture mem- 
ory 126 by rendering control process(or) block 104 in 
the same manner as texture data taken from run-time 
database 110. A shared memory arrangement can be 
used to avoid bottlenecks in the command stream be- 
tween blocks 104 and 106. Alternatively, dynamically 
generated texture data may be fed directly into texture 
memory 126 using an alternative path, such as a 
dedicated data bus coupling texture memory 126 to 
an external texture data source. 

Further, these texture data can be generated or 
modified at any time prior to their transfer into texture 
memory 126. If desired, such textures can be gener- 
ated or modified in real time. Thus, the texture data 
which are transferred into a particular position within 
texture memory 126 (and thereby associated anyone 
or more geometry surfaces which reference that pos- 
ition) in accordance with principles of the present in- 
vention can be wholly generated, or modified, in re- 
sponse to an action of the viewer to provide a display 
of one or more interactive texture patterns. The tex- 
ture pattern can be drawn by scenario process(or) 
block 106. I/O process(or) block 108, or a not her com- 
puter, such that dynamically changing computer-gen- 
erated text and/or graphic images, or digitized video, 
can be introduced as textures. 

Thus, when rendering control process(or) block 
1 04 receives a command from scenario process(or) 
block 106 to load a particular texture map into texture 
memory 126, rendering control process(or) block 104 
retrieves the texture map from run-time database 110 
or storage unit 126a in accordance with file reference 
information provided in the command and stores the 
texture map into texture memory 126 at the memory 
location specified in the control header of the texture 
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map. This same memory location is addressed (di- 
rectly, or indirectly using the above-mentioned look- 
up table) by the texture memory pointers) of the poly- 
gons) to which the texture is to be applied, as shown 
for example by arrow 210 in FIG. 2. (Arrow 210 illus- 
trates a texture memory pointer in a polygon definition 
within object geometry 202 pointing to an address in 
texture memory 126 corresponding to the origin of a 
rectangular (u,v) texture coordinate space 212). 

In FIG. 2, a plurality of time- sequenced textures 
stored in run-time database 110 are illustrated as tex- 
tures tO, t1, t2, t3 ... tn. Depending on the type of tex- 
ture being animated (e.g. the number of colors in the 
texture), each of textures t0...tn may include one or 
more texture maps. For example, in the preferred em- 
bodiment each of textures t0...tn is composed of four 
maps: red, green, blue, and alpha (transparency). A 
plurality of different texture animations can be run 
concurrently in system 1 00 by assigning different sets 
of time-sequenced textures to different areas in tex- 
ture memory 126. Rendering control process(or) 
block 104 maintains a memory-resident table 214 of 
control information relating to each pending texture 
animation. Rendering control process(or) block 104 
uses this-table to keep track of the animations. As 
shown in FIG. 2, table 214 includes information such 
as: the length of the animation (e.g., the number of 
images in the animation sequence), the type of ani- 
mation (e.g., whether the animation is a continuous 
loop which runs from beginning to end and then re- 
peats, or a reversing sequence which runs from be- 
ginning to end, end to beginning, etc.), the current 
position in the animation (e.g., the image in the se- 
quence currently being displayed), etc. In addition, as 
shown in FIG. 2, table 214 also includes texture file 
pointers (such as 218) and pointers into the texture 
memory (such as 220 and 222) correlating animated 
texture file addresses in run-time database 110 with 
respectively assigned areas at1, at2, etc. in texture 
memory 126. 

Prior to starting an animation, rendering control 
process(or) block 104 receives a command from sce- 
nario process(or) block 106 instructing rendering con- 
trol process(or) block 104 to load a particular anima- 
tion sequence into a selected areas in texture mem- 
ory 126. The command is generated by scenario proc- 
essor) block 106, and is issued responsive to an exe- 
cuted instruction of the control program in its main 
memory. The command identifies the beginning ad- 
dress of each file in t he run-time database comprising 
the desired animation sequence. This information is 
provided to scenario process(or) block 1 06 when, as 
part of the process of loading run-time database 110, 
rendering control process(or) block 104 emits the file 
references and location data for all texture files in run- 
time database 110 or storage unit 126a to scenario 
process(or) block 106. Responsive to this command, 
rendering control process(or) block 104 loads the da- 



tabase file address(es) into the proper pointer loca- 
tion^) within table 214. 

In response to a subsequent command from sce- 
nario process(or) block 106 instructing rendering con- 
5 trol process(or) block 104 to start the animation 
(which command may immediately follow the load 
command), rendering control process(or) block 104 
transfers the first texture map from run-time database 
110 or storage unit 126a into texture memory 126 as 
10 shown by dotted line 224. Assuming that an object 
data structure stored in memory 120 has been as- 
signed an object geometry which includes a polygon 
that points to the area of texture memory 126 into 
which the first texture has been loaded, and that the 
15 position and orientation of the polygon is such that it 
is within the viewer's field of view, rendering pipeline 
102 renders the polygon with the first texture of the 
animation in the same manner as it would a conven- 
tional texture. Thereafter, rendering control proc- 
20 ess(or) block 1 04 automatically transfers subsequent 
texture maps in the animation sequence from run-ti- 
me database 110 or storage unit 126a into the same 
area of texture memory 126, and the polygon is ren- 
dered with each such texture. This process can be re- 
25 peated to display the entire animation sequence, or 
any desired portion of that sequence. The rate at 
which the textures are transferred to texture memory 
126, and thus the rate at which they are rendered, is 
chosen to provide smooth animation. As discussed 
30 below, this rate typically need be only a fraction of the 
update rate of the graphics subsystem. For example, 
in a preferred embodiment the texture is changed at 
a 10 Hz rate — i.e., once every six times that the 
graphics subsystem redraws the image of the poly- 
35 gon on which the texture is displayed. 

Using the above method, animated textures can 
be applied to substantially any displayed polygon in a 
scene, including polygons forming surfaces of static 
or moving objects. With careful and coordinated use 
40 of conventional object management and animation 
choreography techniques, scenario process(or) block 
106 can be caused to synchronize the motion and at- 
titude of a moving object with an animated texture dis- 
played on one or more of its surfaces. 
45 In lieu of, or in addition to, the above-described 

technique of sequentially transferring texture maps 
into a designated area in texture memory, other tech- 
niques can be used to provide texture animation in ac- 
cordance with principles of the present invention. For 
so example, although it is convenient to implement tex- 
ture animation by sequentially transferring time- 
sequenced textures into the same area in texture 
memory using a particular texture memory address 
as a common reference point for aligning the u,v co- 
55 ordinate spaces of the time-sequenced textures, it is 
not necessary that time-sequenced textures within a 
set be stored in the same area in texture memory 126. 
If desired, time-sequenced textures can be sequen- 
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tially transferred into different areas in texture mem- 
ory as long as the different areas are appropriately 
associated at render time with the surface on which 
the animation is to be rendered. Such flexibility can 
be added to system 100 by having rendering control 
process(or) block 104 or scenario process(or) block 
1 06 change the texture memory address value of the 
appropriate polygon texture memory pointer during 
run-time to reflect a change in the texture memory 
area in which a time-sequenced texture to be applied 
to the polygon has been stored. Alternatively, if a tex- 
ture memory pointer references indirectly through a 
memory-resident look-up table as previously descri- 
bed, the polygon definitions pointer can be left un- 
changed and the texture memory address in the table 
can be changed appropriately. 

Also, given sufficient texture memory size, the 
textures comprising a time-sequenced texture set can 
be stored concurrently in different areas of texture 
memory instead of being sequentially transferred into 
texture memory. In this case, animation is provided 
by periodically changing the area of texture memory 
addressed by the texture memory pointer of a poly- 
gon to render the polygon with the textures of the set 
in time-sequenced order, either by changing the val- 
ue of the pointer itself (if the pointer directly address- 
es the texture memory) or a memory-resident index 
table to which the pointer refers. 

In another exemplary embodiment, texture ani- 
mation is achieved using pointer 210 by periodically 
offsetting the mapping of a polygon geometry to the 
u,v coordinate space of a texture map containing a 
plurality of time-sequenced images within the same 
texture map. More particularly, an area of texture 
memory 126 may be pre-loaded with a number of 
time-sequenced image frames tiled into a single u.v 
texture space. When rendering a polygon associated 
with this texture space, the vertices of the polygon 
are transformed from object space coordinates into 
texture space coordinates by indexing an offset 
(du.dv) in the u.v coordinate system. This indexing 
preferably executes a raster pattern by indexing in a 
row-wise or column-wise sequence. 

If access to pinter 210 is unavailable in a partic- 
ular image generator or the handling of pointers to 
change textures is not fast enough for some architec- 
tural reason, the invention still may be implemented 
by manipulation of pointer 216. In this alternative em- 
bodiment of the invention, texture animation can be 
implemented by associating a polygon of different 
ones of object geometries 202, 204, 206 etc. with dif- 
ferent ones of time-sequenced textures tO, t1, t2, etc. 
loaded into texture memory 126. To achieve texture 
animation with this embodiment, pointer 216 would 
be used to sequentially point an object in the object 
list maintained by memory 1 20 to different ones of ob- 
ject geometries 202, 204, 206 etc. Different polygons 
would thus be sequentially rendered, each having a 



different applied one of the time-sequence of tex- 
tures. The net effect would be that a plurality of time- 
sequenced textures would be displayed, the textures 
mapped onto different polygons, to create the texture 

5 animation. 

Yet another way to provide texture animation, in 
the event that pointer 216 is not available or cannot 
be manipulated fast enough, is to identify a plurality 
of objects in the object list maintained by memory 120 

10 each associated by a pointer to a different one of ob- 
ject geometries 202, 204, 206, etc. each of which, in 
turn, has a polygon associated with a different one of 
textures tO, t1, t2, etc. loaded into texture memory 
1 26. To display a texture animation sequence, the se- 

15 quence of objects would be rendered at the appropri- 
ate texture animation rate. As with the previously de- 
scribed embodiment, the net result would be that dif- 
ferent polygons would be sequentially rendered, each 
having a different applied one of the time-sequence 

20 of textures, to create the texture animation. 

In addition, texture animation can be implement- 
ed such that only a portion of a texture is time- 
sequenced. For example, instead of sequentially 
transferring a set of full time-sequenced textures into 

25 a particular texture space, smaller time-sequenced 
textures can be sequentially transferred into a given 
portion of that space to overwrite only a section of a 
larger texture residing there. The small textures can 
be aligned by associating them with a common refer- 

30 ence point within the larger texture. Thus, any partic- 
ular section of a texture can be selectively animated. 
Further, responsive to a viewer action detected by 
viewer head sensor 116 or viewer activity sensors 
118, scenario process(or) block 106 can command 

35 rendering control process(or) block 104 to switch be- 
tween different sets of stored or dynamically gener- 
ated time-sequenced textures when sequentially 
transferring textures into a particular texture space in 
texture memory 126, thus providing interactive ani- 

40 mation. 

The sourcing and generation of time-sequenced 
textures are illustrated by FIGS. 3 and 4. 

FIG. 3 shows an exemplary method for introduc- 
ing realistically animated characters into a virtual 

45 world by scripting and recording an event or act per- 
formed by a person in the real world which, in accor- 
dance with principles of the present invention, is to be 
re-enacted in the virtual world by a photo- realistic 
likeness of the actor. Assume that a scene in the vir- 

50 tual world requires a man and a woman to enter a 
square room each through a different doorway locat- 
ed respectively midway along adjacent walls of the 
room. The viewpoint of the person who will be viewing 
the scene in the virtual world is known, or assumed, 

55 to be at a specific position in, or in a general area of, 
the room. Assume also that the man is to walk into the 
room toward its center, and the woman is to walk in 
an arc around the room toward the doorway through 



10 



19 



EP 0 583 061 A2 



20 



which the man entered. The analog of these events 
are performed in the real world by male and female 
actors 302 and 304 before video cameras 306 and 
308. Realism in the virtual world is best, and prefer- 
ably, achieved by properly scaling the movements of 
the human actors at the time of recording to the di- 
mensions of the virtual world scene into which the 
animated likenesses of the actors are to be inserted 
as textures. 

Accordingly, as shown in FIG. 3, the actors are 
provided with markers, such as tape on the floor, de- 
lineating the positions of doors and walls of a room di- 
mensioned in accordance with the dimensions of the 
room to be modeled in the virtual world. Such mark- 
ers may be provided in other positions as well, or the 
ceiling lor example, and preferably outside of the view 
of the cameras. A variety of backgrounds also may be 
used. If a is desired that the texture to be displayed 
in the virtual world is the likeness of an actor against 
a particular background scene that cannot easily be 
modeled in the virtual world, the background scene 
can be incorporated into the texture by representing 
the background scene on a wall 310 behind the actor 
when recording the actor's performance by cinemat- 
ography or videography. Alternatively, the wall may 
be a conventional "blue screen" of the type used in or- 
dinary video production to provide a background that 
can be changed in a video switching process com- 
monly referred to as chroma-keying. This process is 
often used in broadcast television, for example in 
news weather reports, to continuously select which of 
two synchronized video inputs will be used, typically 
based on the chrominance signal of one of those in- 
puts. In this manner, a blue field (or other constant 
color) behind the weather man is replaced by a sat- 
ellite map in real time. An analogous process used in 
film production and photography is commonly refer- 
red to as blue-screening, whereby an actor is filmed 
or photographed in front of a spectrally pure color 
screen, and is later silhouetted and composited into 
another filmed scene or photograph. 

The chroma-key process, or an analogous chro- 
minance-based or luminance- based digital image 
processing technique, can be used in accordance 
with the present invention to generate animated tex- 
tures in which selected portions of the texture, such 
as a background behind the likeness of a person, are 
made transparent when overlaid on a polygon sur- 
face, so that the portions of the surface on which the 
transparent texture is overlaid are not displayed to the 
viewer. This is discussed more fully below. 

When recording an actor, the size of the actor 
should preferably be kept substantially constant in the 
eye of the recording camera so that the recorded like- 
ness of the actor will not change size unreal tstically 
when a time-sequenced set of textures are generated 
from the recording. To the extent t hat a viewer should 
perceive a change in an animated character's size 



corresponding to a change in distance in the virtual 
world between the animated character and the view- 
er, this change in size will be provided by the render- 
ing pipeline when rendering an image of the actor. 
5 To provide an animated character whose image 

appears properly oriented to the viewer, the actor pre- 
ferably should be filmed from a direction analogous to 
the anticipated direction of the viewer in the virtual 
world. Thus, if the anticipated position of the viewer's 
10 viewpoint in the virtual world analog of the room 
boundary 320 shown in FIG. 3 corresponded to point 
312, camera 306 would preferably be positioned at 
that point, or along the line of the actor's trajectory, 
when recording entry of male actor 302 into the room. 
15 As the actor moves closer to the camera, the size of 
the actor can be kept substantially constant by back- 
ing the camera up, or by dynamically zooming the 
camera. Alternatively, conventional video post- 
production techniques can be used to normalize the 
20 size of the actor. Likewise, to provide a properly ori- 
ented image of female actor 304 as she crosses the 
room along arc 314, the recording camera (e.g., cam- 
era 308) preferably should be placed at point 312 at 
the center of curvature of arc 314, to pan in synchron- 
25 ism with the actor's movements from that point. 

Multiple cameras can be used to record the same 
actor from different angles. Using this technique, a 
plurality of sets of time-sequenced textures can be 
generated to provide a realistic image of the actor to 
30 the viewer from different viewpoints. The appropriate 
texture sequence can be selected based on the rela- 
tive locations of the viewer and the animated charac- 
ter in the virtual world, and the orientation of the sur- 
face to which the texture is applied can be kept nor- 
35 mal to the viewer, thereby dynamically changing both 
the animation and the orientation of the surface to 
disguise the 2-D nature of the animation. 

If desired, props and furniture can be provided for 
the actors incorporated into the texture. For instance, 
40 an actor can be placed behind a counter from behind 
which he pulls a gun. Such props can recorded for lat- 
er visual display in the virtual world with the texture 
of the actor, or the prop (e.g., the counter) can be col- 
ored so as to be part of the blue-screen background. 
45 By blue-screen coloring the counter, the texture of the 
recorded actor's torso will be truncated so that his im- 
age can be realistically displayed in the virtual world 
pulling a gun from behind a virtual counter modeled 
and generated by the computer system. 
so In addition, matched lighting and shadows can be 

provided as part of the texture to reinforce the illusion 
of a 3-D character in the virtual world. For example, 
in cases where the lighting situation into which an ani- 
mated texture is to be inserted is known, the source 
55 imagery for the animated texture can be created with 
that in mind. If a harsh midday sun is to be simulated 
in the virtual world, t hen an actor might be filmed with 
a lighting setup that effectively emulates that lighting 
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condition. Alternatively, if the final intent is a scene lit 
by a burning building, then an actor might be lit by dif- 
fuse, flickering red and yellow lighting. All elements of 
intensity, contrast, color, modulation, distribution and 
direction of light sources illuminating the actors during 5 
recording may be manipulated to enhance the blend- 
ing of the animated texture into a computer synthe- 
sized scene. 

An animated texture character can be caused to 
cast a shadow in the virtual world, having a shape 10 
consistent with the shape of a shadow that would be 
thrown by a 3-D being in the real world, by incorpor- 
ating an actor's real-world shadow into the actor's 
texture. This can be accomplished, in an exemplary 
fashion, by placing in the real-world room various is 
models 316 and 318 of objects (such as furniture or 
walls) that will appear at like positions in the analog 
room in the virtual world. The model furniture itself 
need not be incorporated into the texture. The model 
furniture can be covered in blue screen material when 20 
the recording of the actor is made, which captures the 
actor s shadow on the model furniture. Known video 
processing techniques can be used to key on the 
chromaticity and luminosity of the blue-screen mate- 
rial to filter from t he texture all but actor and t he shad- 25 
ow. A shadow might be represented in a texture map 
as a biack field of some partial transparency. When 
rendered, the shadow will reduce the luminosity of 
any portion of an object it obscures. 

Another improvement is possible in cases where 30 
the animation does not result in significant movement 
of key points in the texture image (as in the case of a 
face which talks, but does not tilt, shake, or nod; or as 
in the case of a simulated screen of a television set). 
The geometry which underlies the animated texture 35 
in these cases does not need to be planar. For in- 
stance, 1 0 to 30 polygons which need not move rela- 
tive to one another might be used to "sculpt" a virtual 
world face having a 3-D shape approximating the 3- 
D shape of a face of a person existing in the real 40 
world. An an imated texture of t hat or a not her person's 
face could thenbeplacedonthis geometric surface, 
with care taken to align the texture image of the nose 
with the sculpted geometry of the nose, chin with 
chin, etc. A conventional projection type mapping al- 45 
gorithm can be used to assign (u,v) texture coordin- 
ates from the animated texture to the vertices of the 
polygons. The character, when viewed straight on, 
would present the image of the texture as recorded. 
From the side, a 3-D profile -- rather than a foreshort- so 
ened, flat face-- would be seen. Different views of the 
face thus would be rendered as a function of the view- 
point from which the face is viewed to create an illu- 
sion of three-dimensionality of the face in the virtual 
world. 55 

In accordance with another aspect of the present 
invention, an animated character may be created us- 
ing a hybrid of texture animation and conventional 



model animation techniques. Indeed, in certain cir- 
cumstances such a hybrid approach results in the 
most effective use of texture animation. For example, 
a character may be composed of a body and a head, 
with the body being modeled by one or more conven- 
tionally textured polygons and the head being mod- 
eled by one or more polygons to which an animated 
texture of a face is applied. By texture-animating only 
the face, the resolution of the texture can be made 
greater, thus allowing the character's facial expres- 
sions to be displayed in greater detail than would be 
the case if a simBarly-sized animated texture were 
used to display the entire character. This is but one of 
many possible examples of how t he texture animation 
technique of the present invention can be used in a 
selective manner to animate realistically a portion of 
a virtual world character or other entity. As another 
example, if a character is to perform a hand gesture 
(e.g., a magician's hand performing a card trick), the 
magician's hand might be texture-animated while the 
remaining portions of his body are modeled and tex- 
tured in a conventional manner. The animated image 
may comprise only a portion of a larger texture ap- 
plied to a surface, such that if a single texture is used 
to image a face, selective portions of the face, such 
as the eyes or mouth, can be animated independent- 
ly. 

In addition, animated textures can be used in a 
stereoscopic display. Texture video would be cap- 
tured in 3-D by two cameras. Two texture animations 
would be prepared, one from each camera. When an 
image is rendered, the right camera derived texture is 
used for the display for the right eye, and the other 
for the left. When displayed in the virtual world, there 
would be a particular distance (the real world dis- 
tance at which the texture was recorded divided by ra- 
tio of the camera interocular to the viewer's interocu- 
lar) at which the convergence would be substantially 
correct and the 3-D illusion optimized. To minimize 
animated texture update bandwidth when viewing at 
some long range where the stereoscopic effect is 
minimal, the two separate textures can be abandoned 
for either the right or left singly. 

In lieu of a recording of a live being or other ob- 
ject, and further in accordance with the present in- 
vention, source imagery for an animated texture can 
be derived from sequences of cartoon or other draw- 
ings. The source imagery for such drawings can be 
generated by substantially any process. For instance, 
the drawings could be derived from free-hand artist's 
sketches or digitized images drawn using a computer- 
ized "paint box" or personal computer system running 
a drawing or CAD program. However created, real- 
time animated stylistic characters (e.g., cartoon fig- 
ures) can be introduced using the present invention 
into a computer graphic virtual world to add to the lev- 
el of entertainment enjoyed by the viewer in a way 
that would be inordinately expensive to accomplish 
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using conventional 3-D computer modeling schemes. 

Texture animation can make use of substantially 
any conventional type of texture map available in a 
graphics subsystem. In a preferred embodiment, ani- 
mated textures are generated using full color texture 
maps (i.e. maps def ining red, green and blue primary 
color components for each texel) in conjunction with 
an alpha "channel" or "plane" to designate a degree 
of transparency of the texel. The alpha channel allows 
for effects such as silhouetting, shadows, smokey 
characters, and dirty window panes. 

The choice of texture map type used for anima- 
tion in a particular computer graphic system may de- 
pend on the capabilities of the particular graphics 
subsystem used. For example, the architecture of the 
graphics subsystem may constrain the maximum flow 
of texture data into the rendering pipeline, thus nec- 
essitating a trade-off among the number of simultane- 
ous animations, animation size, rate, channels/texel 
and bits/channel, and possibly other capabilities of 
the graphics subsystems. In principle, therefore, a 
texture map type that provides an acceptable aes- 
thetic effect with the lowest amount of texture data 
possible usually should be selected. 

The frequency at which an animated texture is up- 
dated (i.e. changed from one time-sequenced texture 
to the next) need not match the frequency at which a 
real time computer graphic system updates the sur- 
rounding graphic display. For example, a convention- 
al 3-D graphic subsystems may refresh a graphic dis- 
play at a rate of 50 Hz or 60 Hz, and may update the 
display at a rate equal to the refresh rate. Such a high 
update rate is important to providing high speed 
movement in the virtual world and to avoiding viewer 
discomfort (a perceived lag between the viewer's 
head movement and the changing scene can cause 
nausea). However, this high update rate is more than 
necessary to provide quality animation. An animation 
update rate of about 5-10 Hz has been found to pro- 
vide acceptable results in most cases. As in the art of 
cartoon animation, the number of different images 
needed to be generated per second depends on the 
nature of the action. Slow gestures might be welt 
shown by updating a texture at only a 5 Hz rate, while 
more rapid gestures might require that texture be up- 
dated at a 10-15 Hz rate, or possibly at a rate as high 
as 20 Hz to 30 Hz for extremely rapid or complex mo- 
tion. 

Neither is the resolution of the animated texture 
required to match that of the graphic display. Suitable 
animation, for example, may be achieved using four 
channel texture maps (e.g., red, green, blue and al- 
pha) having 128x128 texels with 4-bit planes (i.e., 4 
bits per texel per channel). Alternatively, even fewer 
bits per texel could be satisfactory rf interpreted in a 
conventional manner as an index into a color lookup 
table. In determining the appropriate resolution of the 
texture, it is necessary to consider the likely relative 



size of the animated figure with respect to the overall 
display. If the character wilt always be small (filling 
less than a fraction of the screen), a low resolution 
texture can be used to avoid wasting "texture band- 

5 width," a term used herein to refer generally to the ca- 
pacity of a real time computer graphic system to 
change textures when rendering 3-D images. This is 
no different than the resolution considerations need- 
ed in conventional non- animated texture maps. 

10 Additionally, an animated texture may be created 

using a time-sequenced set of conventional "MIP" 
maps. MIP maps are texture maps which contain mul- 
tiple versions of a texture image that have been suc- 
cessively filtered to have lower spatial frequency, 

15 thereby requiring fewer texels and providing a mini- 
fied representation without visual aliasing. Lower-lev- 
el-of-detail portions of the map can be progressively 
substituted for higher-level-of-detail portions as the 
textured surface extends or moves into the back- 

20 ground, thus minimizing aliasing in the texture as it 
becomes smaller. In accordance with conventional 
techniques, the relative size of the texels in the gen- 
erated image can be used to determine which two lev- 
els of the MIP map to use for interpolation of any given 

25 pixel. 

FIG. 4 illustrates an exemplary method for gen- 
erating a time-sequenced set of textures in accor- 
dance with one aspect of the present invention. A vid- 
eo recording of a live actor in a scripted performance 

30 is generated by camera 400, and the analog video is 
captured on an analog or digital video cassette recor- 
der/player 402. Camera 400 may be any type of video 
camera, although for high-quality textures it is prefer- 
red that the camera be an industrial grade, high reso- 

35 lution, three-detector, charge-coupled-device (CCD) 
camera providing an S-video output. It is further pre- 
ferred that video cassette recorder/player 402 cap- 
ture sound simultaneously with video for later use in 
synchronizing the animated texture of the actor with 

40 a reproduced voice or other sound generated by the 
actor. A conventional S-VHS video cassette recor- 
der/player is suitable for this purpose. 

To better separate the frames of t he video record- 
ing for subsequent digitization, the video recording is 

45 played back and re-recorded on a conventional video 
disk recorder/player 404. Frames of the video record- 
ing are then selectively digitized by a conventional 
digitizer card in a conventional general purpose com- 
puter 406. For this purpose, a VidPix model digitizer 

so card installed in a SPARCstation 2 workstation, both 
commercially available from Sun Microsystems Inc., 
of Mountain View, California, may be used. The com- 
puter automates and synchronizes the advance of 
frames on the video disk recorder/player using a 

55 frame control signal output connected by an RS-232 
port to the frame advance input of the video disk re- 
corder/player. The frame rate (30 Hz) of the video typ- 
ically will exceed that required for acceptable anima- 

13 
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tion (which may, as previously explained, only require 
an update rate of about 5-10 Hz). Accordingly, in such 
a case only every other or third frame (or some other 
skip-frame count) is used. Skipping frames can serve 
another purpose as well. Some interlaced video sys- 5 
terns divide frames into multiple fields which, if cap- 
tured at different times, may introduce some unaes- 
thetic artifacts, in which case the effect of the arti- 
facts may be mitigated by selecting only data from odd 
or even fields. 70 

The digitizer of process(or) 406 scans each se- 
lected frame and divides it into an array of pixels. The 
digitizer computes three scalar digital values indica- 
tive respectively of the luminance (or intensity) of the 
red. green and blue color components of each pixel, 15 
thus creating three arrays (or "channels" or "planes") 
of digital data for each frame. The digitized frames are 
then stored in time-sequence order as sequential R- 
G-Bdata f ties 408. meaning that a separate file is cre- 
ated for each frame, wit ht he R-G-B data cor respond- 20 
ing to the first selected frame being grouped together 
as contiguous arrays and stored as the first file of the 
sequence, the R-G-B data for the next selected frame 
being grouped together and stored as a second file 
following the first, and so on. 25 

To mark selected portions of the content of each 
frame, or to make all or selected portions of each 
frame translucent (e.g., to represent a ghost or to fade 
the texture) or wholly transparent, the R-G-B data for 
each frame are processed by an alpha (a) plane gen- 30 
erator410. Alpha plane generator 410 passes the R- 
G-B data through a series of filters to generate a 
fourth channel or plane which specifies an alpha val- 
ue, or code, for each pixel as a function of the lumi- 
nance and/or chrominance of the pixel. The filtering 35 
process is accomplished in the digital domain by soft- 
ware running on a general purpose computer, such as 
the above-mentioned Sun Microsystems workstation, 
using well-known linear transform routines to derive 
a luminance and/or chrominance-based value for 40 
each pixel, which is then mapped to a luminance-key- 
ed and/or chroma-keyed color space. For example, to 
provide basic silhouetting, an area in color space may 
be designated as "background," and any pixels hav- 
ing a computed chromaticity and/or luminosity value 45 
within that area are assigned an alpha value indica- 
tive of full transparency. Those pixels falling within an 
area in color space not classified as background are 
assigned an alpha value indicative of full or partial 
opacity. The result is that a fourth digital data array so 
is created for each digitized frame, providing each pix- 
el in a frame with an associated degree of transpar- 
ency keyed to its chromaticity or luminosity. 

The alpha data are added to each of the R-G-B 
data files, which are then combined and stored again 55 
by the workstation in time-sequence order as se- 
quential R-G-B-a data files 412 (which may be sepa- 
rate files or a single integrated file). The workstation 



then reconfigures the format of the files to provide 
four separate animated texture files, each file con- 
taining one channel of each frame in the animation ar- 
ranged in time-sequenced order. Thus, for example, 
one file would contain contiguous arrays of digital 
data indicative of the red component of the frames in 
the animation, in time-sequence order. Another file 
would contain contiguous arrays of digital data indica- 
tive of the blue component of the frames in the ani- 
mation, in the same time-sequence order, and so on 
for the green and alpha channels. The workstation at- 
taches a header to each file which includes a file 
name (identifying all four files of an R-G-B-a animat- 
ed texture by the same name), and parametric infor- 
mation concerning the dimensions of the file, such as 
the height and width (in texels) and number of frames 
in the file. 

When an animated (or conventional) texture hav- 
ing an alpha channel is rendered by a computer 
graphics system, each alpha code in the texture map 
is interpreted by the rendering system as specifying 
the degree to which the portion of the surface to 
which the corresponding texel is applied is transpar- 
ent. Thus, the alpha data specify whether that portion 
of the surface, if displayed, should fully occlude, only 
partially obscure, or not occlude or obscure in any 
way, whatever background (e.g., terrain or other ob- 
jects) is behind it. 

Animated texture files can be included into the 
run-time database of the computer graphics system, 
along with geometry/model files 414 (which includes 
ordinary textures), by a conventional compiler pro- 
gram 416. Although conventional 3-D graphic sys- 
tems often have dedicated database developmental 
tools which can be used to generate textures for use 
with the present invention, a virtual reality system de- 
signer may choose to develop additional database 
tools to assist in creation of the run-time database, in- 
cluding any translation routines necessary to gener- 
ate data files in the format required by the computer 
graphic system. Such tools could be readily devel- 
oped, using known, conventional techniques, by per- 
sons of ordinary skill in the art. 

Preferably, the animated texture files are then 
compressed in a conventional manner (e.g., by run- 
length encoding, or by conventional JPEG compres- 
sion as shown in FIG. 4, item 418) to reduce the 
amount of data that must be stored and transferred in 
and out of secondary storage. When the animated 
texture files are retrieved from run-time database 
110, a complementary decompression process is 
used to expand the texture data. (Note that the com- 
pression/decompression process need not be loss- 
less.) 

Various alternatives to the above-described 
methodology are available for creating animated tex- 
ture files in accordance with the present invention. 
For example, a specialized type of video processing 
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device conventionally used in television broadcasts, 
called a "Video-matter 0 and commercially available 
from Ultimatte of Chatsworth, California, can be con- 
nected between camera 400 and video cassette re- 
corder/player 402 to convert background colors in an 5 
analog video signal to ultra-black, thus making it eas- 
ier to subsequently generate an alpha plane which sil- 
houettes a character. As another example, a comput- 
er having a video frame grabber/digitizer card, such 
as an Apple Macintosh computer, can be connected 10 
directly to camera 400 to capture frames of live video, 
in lieu of using recorders 402 and 404. The captured 
video can be edited and converted into animated tex- 
ture files using software programs such as Adobe 
PhotoShop and Adobe Premier, commercially avail- 15 
able from Adobe Systems, Inc. Customized plug-in 
program modules would be required with these pro- 
grams to implement an alpha plane generator and to 
store the R-G-B-afile in the proper sequential format, 
which readily can be developed by persons of ordi- 20 
nary skill in the art 

As another alternative, the functions of the video 
frame grabber/digitizer, alpha channel generator and 
f ile formatter can be integrated into a high-speed real 
lime system which transfers images from camera 400 25 
directly into a video texture buffer of the interactive 3- 
D computer graphics system for immediate or delayed 
use as an animated texture. Using such a technique, 
or the previously described alternatives, a texture can 
be generated which portrays an animated photo- 1 ike- 30 
ness of the viewer in the virtual world. 

It will be appreciated by those skilled in the art 
that the present invention can be practiced and imple- 
mented by other than the described embodiments, 
which are presented for purposes of illustration and 35 
not of limitation, and the present invention is limited 
only by the claims which follow: 



Claims 



resentative of the plurality of other surfaces, and 
a sequence of images based on the data repre- 
sentative of the first surface and the data repre- 
sentative of subsequent ones of said succession 
of textures, to display the figure in action in the 
virtual world on at least a portion of the first sur- 
face as texture which changes at intervals over 
a period of time so that the figure appears ani- 
mated. 

2. The method of Claim 1 , wherein the data repre- 
sentative of the first surface are three- 
dimensional data. 

3. The method of Claim 1, further comprising the 
steps of: 

storing data representative of a second 
surface; 

storing data derived from a succession of 
at least three temporally related source images 
depicting a second figure in action at different 
moments in time in the form of a second succes- 
sion of at least three time-se que need textures; 
and 

rendering in real time a sequence of im- 
ages based on the data representative of the 
second surface and the data representative of 
subsequent ones of said second succession of 
textures to display the second figure in action on 
at least a portion of the second surface as tex- 
ture which changes at intervals over a period of 
time so that the second figure appears animat- 
ed, 

whereby two texture-animated figures in 
action are displayed in the virtual world. 



4. A method for animating a rendition of a figure in 
action at a position in a three-dimensional virtual 
world generated by a three-dimensional comput- 
40 er graphic system, the method comprising the 

steps of: 

storing data representative of a first sur- 
face; 

storing data derived from a succession of 
45 at least three temporally related source images 

depicting the figure in action at different mo- 
ments in time in the form of a succession of at 
least three time-sequenced textures; 

associating the first surface to a texture 
50 memory; 

sequentially transferring to said texture 
memory the stored texture data for subsequent 
ones of said succession of time-sequenced tex- 
tures; 

55 storing three-dimensional data represen- 

tative of a plurality of other surfaces forming the 
virtual world; and 

rendering in real time images of the virtual 



1. A method for animating a rendition of a figure in 
action at a position in a three-dimensional virtual 
world generated by a three-dimensional comput- 
er graphic system, the method comprising the 
steps of: 

storing data representative of a first sur- 
face; 

storing data derived from a succession of 
at least three temporally related source images 
depicting the figure in action at different mo- 
ments in time in the form of a succession of at 
least three time-sequenced textures; 

storing three-dimensional data represen- 
tee of a plurality of other surfaces forming the 
virtual world; and 

rendering in real time images of the virtual 
world based on the three-dimensional data rep- 
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world based on the three-dimensional data rep- 
resentative of the plurality of other surfaces, and 
a sequence of images based on the stored data 
representative of the first surface and the se- 
quentially transferred texture data in said texture 
memory, to display the figure in action in the vir- 
tual world on at least a portion of the first surface 
as texture which changes at intervals over a per- 
iod of time so that the figure appears animated. 



ages based on the stored data representative of 
the second surface and the sequentially trans- 
ferred second texture data in said texture mem- 
ory to display the second figure in action on at 
least a portion of the second surface as texture 
which changes at intervals over a period of time 
so that the second figure appears animated, 

whereby two texture-animated figures in 
action are displayed in the virtual world. 



the succession of textures derives from a 
succession of at least three temporally related 
drawings. 

5 11. The method of Claim 10, wherein the drawings 
comprise cartoon drawings and the texture dis- 
played appears to be an animated cartoon. 

12. A method for animating a rendition of a figure in 
action on an object situated in a three- 
dimensional virtual world generated by a com- 
puter graphic system, the object including a plur- 
ality of surfaces, and the rendition capable of be- 
ing displayed from a plurality of viewpoints look- 
ing into the virtual world from specific positions 
and orientations, the method comprising the 
steps of: 

storing three-dimensional data represen- 
tative of the surfaces of the object; 

storing data derived from a succession of 
at least three temporally related source images 
depicting the figure in action at different mo- 
ments in time in the form of a succession of at 
least three time-sequenced textures; and 

rendering in real time a sequence of im- 
ages based on the data representative of the 
surfaces and the data representative of subse- 
quent ones of said succession of textures to dis- 
play the object relative to different ones of said 
plurality of viewpoints with texture applied to at 
least a portion of at least one of the surfaces with 
the texture changing at intervals over a period of 
time so that the figure appears animated. 

35 13. The method of Claim 12, wherein texture is ap- 
plied over the plurality of surfaces with the tex- 
ture over those surfaces changing with time so 
as to appear animated. 

14. The method of Claim 13, wherein: 

the succession of time-sequenced tex- 
tures derives from a recorded succession of at 
least three temporally related source images de- 
picting at least a portion of an entity existing in 
the real world; and 

the plurality of surfaces model an object 
having a three-dimensional shape approximat- 
ing the three-dimensional shape of at least the 
portion of the entity, 

whereby different perspective views of 
the texture can be displayed for different ones of 
the plurality of viewpoints such that the object 
displayed appears to be an animated three- 
dimensional likeness of at least that portion of 
the entity. 



40 

7. The method of Claim 3 or 6, wherein the two tex- 
ture-animated figures in action are displayed si- 
multaneously. 

8. The method of Claim 3 or 7 when appended to 45 
Claim 3, wherein the data representative of the 
first and second surfaces are three-dimensional 
data. 

9. The method of Claim 1 or 4, wherein: so 

the succession of time-sequenced tex- 
tures are derived from a recorded succession of 
at least three temporally related source images 
depicting an entity existing in the real world; and 

the texture displayed appears to be an 55 
animated likeness of the entity. 



10. The method of Claim 1 or 4, wherein: 
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5. The method of Claim 4, wherein the first surface 
is associated with a predetermined location in 
said texture memory. 

6. The method of Claim 4, further comprising the 15 
steps of: 

storing data representative of a second 
surface; 

storing data derived from a succession of 
at least three temporally related source images 20 
depicting a second figure in action at different 
moments in time in the form of a second succes- 
sion of at least three time-sequenced textures; 

associating the second surface to said 
texture memory; 25 

sequentially transferring to said texture 
memory the stored texture data for subsequent 
ones of said second succession of time- 
sequenced textures; and 

renderinq in real time a seauence of im- 30 



1 5. A method for use by a computer graphic system 
for rendering a likeness of a real-world living be- 
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ing at a specific three-dimensional position in a 
computer-generated three-dimensional virtual 
world, the rendition capable of being displayed 
from viewpoints having specific positions and 
orientations within the virtual world, the method 5 
comprising the steps of. 

storing three-dimensional data represen- 
tative of at least one surface positioned in the vir- 
tual world; 

storing data representative of a succes- 10 
sion of at least three textures derived from a re- 
corded succession of at least three temporally 
related source images depicting the being at dif- 
ferent moments in time; and 

rendering in real time a sequence of im- is 
ages based on the data representative of the 
surface and the data representative of subse- 
quent ones of said succession of textures to dis- 
play the being relative to different ones of said 
viewpoints as texture applied to at least a portion 20 
of the surface in the virtual world, with the tex- 
ture changing at intervals over a period of time so 
that the being appears animated. 

16. The method Claim 15, wherein: 25 

the three-dimensional data are represen- 
tative of a plurality of surfaces modeling an ob- 
ject positioned in the virtual world, the object 
having a three-dimensional shape approximat- 
ing a three-dimensional shape of the being; and 30 

the changing texture is applied over the 
plurality of surfaces, whereby different perspec- 
tive views of the texture can be displayed for dif- 
ferent ones of the plurality of viewpoints such 
that the object displayed appears to be an ani- 35 
mated three-dimensional likeness of the being. 

17. A method for use by a computer graphic system 
for rendering a likeness of a real-world living be- 
ing in a computer-generated three-dimensional 40 
virtual world, the method comprising the steps 

of: 

staring data representative of a surface 
positioned in the virtual world; 

storing data representative of a plurality of 45 
time-sequenced textures derived from a record- 
ed succession of at least three temporally related 
source images depicting the being at different 
moments in time; and 

rendering in real time a sequence of im- 50 
ages based on the data representative of the 
surface and the data representative of subse- 
quent ones of said succession of textures to dis- 
play the being in the virtual world on at least a 
portion of the surface as texture which changes 55 
at intervals over a period of time so that the being 
appears animated. 



18. A method for animating a figure in action on an 
object against a background in a three- 
dimensional virtual world rendered by a comput- 
er graphic system, the object having at least one 
surface and the rendition capable of being dis- 
played from viewpoints having specific positions 
and orientations within the virtual world, the 
method comprising the steps of: 

storing data representative of the surface; 
storing data derived from a succession of 
at least three temporally related source images 
depicting the figure in action in a scene at differ- 
ent moments in time in the form of a succession 
of at least three time-sequenced textures, said 
data including information identifying a portion of 
texture including at least a portion of the figure 
as texture desired to be visible in the display; 

rendering in real time a sequence of im- 
ages based on the data representative of the 
surface and the data representative of subse- 
quent ones of said succession of textures to dis- 
play relative to different ones of said viewpoints, 
on at least a portion of the surface of the object 
in the virtual world, the texture desired to be visi- 
ble at least obscuring the background and 
changing at intervals over a period of time so that 
the figure appears animated. 

19. The method of Claim 18, wherein the texture de- 
sired to be visible occludes the background. 

20. The method of Claim 18 or 19, wherein the por- 
tion of the time-sequenced textures desired to be 
visible comprises a succession of at least three 
temporally related likenesses of a being existing 
the real world. 

21 . The method of Claim 20, wherein the likenesses 
derive from recorded images of the being. 

22. The method of Claim 15, 17 or 20, wherein the 
being comprises an animal. 

23. A method for use by a computer graphic system 
for rendering the image of a real-world entity sit- 
uated against a background in a computer-gen- 
erated three-dimensional virtual world, the ren- 
dition capable of be ing displayed from viewpoints 
having specific positions and orientations within 
the virtual world, the method comprising the 
steps of: 

storing data representative of a surface; 

storing data derived from a succession of 
at least three temporally related likenesses of 
the entity recorded at different moments in time 
in front of a backdrop as a succession of at least 
three time-sequenced textures, said data includ- 
ing information identif iying the entity to be visible 
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and the backdrop to be invisible in the display; 
and 

rendering in real time a sequence of im- 
ages based on the data representative of the 
surface and the data representative of subse- s 
quent ones of said succession of textures to dis- 
play relative to different ones of said viewpoints, 
as texture on the surface, the likenesses of the 
entity in front of the backdrop, with the displayed 
texture changing at intervals over a period of 10 
time so that the entity appears animated. 

24. The method of Claim 9, 14, 15, 17, 20 or 23, 
wherein the entity or being comprises a person. 

15 

25. The method of Claim 24, wherein the person is 
an actor performing a scripted event 

26. The method of Claim 25, when dependent on 
Claim 21 , wherein the actor performs in front of 20 
a backdrop and the data forming the succession 

of time-sequenced textures includes information 
identifying the backdrop to be not visible in the 
display. 

25 

27. The method of Claim 15, 17, 20 or 23, wherein 
the being is a plant. 

28. A method for animating a figure in action on a 
moving object having at least one surface in a 30 
three-dimensional virtual world rendered by a 
computer graphic system, the rendition capable 

of being displayed from a plurality of viewpoints 
looking into the virtual world from specific posi- 
tions and orientations, the method comprising 35 
the steps of: 

recording a succession of at least three 
temporally related source images depicting an 
entity existing in the real world in front of a back- 
drop, the scripted entity translationally moving 40 
analog to a desired translational movement of 
the object in the virtual world; 

converting a plurality of selected images 
from the recording into data representative of a 
succession of at least three time-sequenced tex- 45 
tures, the texture data including information 
identifying the entity to be visible in the display; 

storing three-dimensional data represen- 
tative of the at least one surface of the moving 
object; and 50 

rendering in real time a sequence of im- 
ages based on the data representative of the 
surface and the data representative of subse- 
quent ones of said succession of textures to dis- 
play relative to different ones of said plurality of 55 
viewpoints the surface of the object textured 
with the likeness of the entity, with the texture 
changing at intervals over a period of time so t hat 



the entity appears animated in synchronism with 
the object's movement. 

29. The method of Claim 28, wherein the recording 
is a video recording. 

30. The method of Claim 28, wherein the recording 
is a film recording. 

31. The method of Claim 30, wherein the recording 
comprises a set of still photographs. 

32. The method of any of Claims 22 or 24 to 28, 
wherein the likeness(es) is a photographic like- 
ness. 

33. The method of Claim 28, wherein: 

the entity is a person; 

the selected images are of the person 
walking; and 

the moving object appears as the person 
walking in synchronism with the person's trans- 
lational movement. 

34. A method for use by a computer graphic system 
for rendering a likeness of a real-world human 
being situated in a computer-generated three- 
dimensional virtual world, the rendition capable 
of being displayed from a plurality of viewpoints 
looking into the virtual world from specific posi- 
tions and orientations, the method comprising 
the steps of: 

storing three-dimensional data represen- 
tative of surfaces forming the virtual world; 

storing data representative of a succes- 
sion of at least three textures derived from a re- 
corded succession of at least three temporally 
related source images depicting the human be- 
ing; and 

rendering in real time images of the virtual 
world based upon the three-dimensional sur- 
face data and a sequence of images of the hu- 
man being based upon the data representative of 
subsequent ones of said succession of textures, 
whereby the human being is displayed in the vir- 
tual world relative to different ones of said plur- 
ality of viewpoints at a specific three- 
dimensional position as texture which changes 
so that the image of the human being appears 
animated. 

35. The method of Claim 34, wherein the human be- 
ing is an actor performing a scripted event. 

36. A method for animating a rendition of a figure in 
action at a position in a three-dimensional virtual 
world generated by a three-dimensional comput- 
er graphic system, the method comprising the 
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steps of: 

storing data representative of a first sur- 
face; 

storing data derived from a succession of 
at least three temporally related source images 
depicting the figure in action at different mo- 
ments in time in the form of a first succession of 
at least three time- sequenced textures; 

storing data derived from a second suc- 
cession of at least three temporally related 
source images depicting the figure in action at 
different moments in time in the form of a second 
succession of at least three time-sequenced tex- 
tures; 

storing three-dimensional data represen- 
tative of a plurality of other surfaces forming the 
virtual world; 

rendering in real time images of the virtual 
world based on the three-dimensional data rep- 
resentative of the plurality of other surfaces, and 
a sequence of images based on the data repre- 
sentative of the first surface and the data repre- 
sentative of subsequent ones of said first suc- 
cession of textures, to display the figure in action 
in the virtual world on at least a portion of the 
first surface as texture which changes at inter- 
vals over a period of time so that the figure ap- 
pears animated; 

detecting an interaction between a viewer 
and the virtual world; and 

in response to a detected interaction be- 
tween the viewer and the virtual world, repeat- 
ing said rendering step based on the data repre- 
sentative of the first surface and the data repre- 
sentative of subsequent ones of said second 
succession of textures, 

whereby the animation of the figure in ac- 
tion is changed in response to the detected inter- 
action between the viewer and the virtual world. 

37. A method for interchanging textures applied to a 
surface situated in a three-dimensional virtual 
world generated by a computer graphics system 
to render a figure with a portion in action, the 
method comprising the steps of: 

storing data representative of the surface; 

storing pattern data representative of the 
figure in the form of a first texture; 

storing pattern data derived from a suc- 
cession of at least three temporally related 
source images depicting an active portion of the 
figure at different moments in time in the form of 
a succession of at least three time-sequenced 
textures; 

associating the data representative of the 
surface to a texture coordinate space in a texture 
memory; 

associating the stored pattern data for t he 



first texture to said texture coordinate space; 

sequentially associating the stored pat- 
tern data for different ones of said succession of 
time-sequenced textures to an area of said tex- 

5 ture coordinate space within said first texture; 

storing three-dimensional data represen- 
tative of a plurality of surfaces forming the virtual 
world; and 

rendering in real time images of the virtual 

10 world based on the three-dimensional data rep- 

resentative of the plurality of surfaces, and a se- 
quence of images based on the stored data rep- 
resentative of the surface and the first and se- 
quentially associated texture patterns, to display 

15 the figure in t he virtual world on at least a portion 

of the surface as texture which changes in part 
at intervals over a period of time so that the ac- 
tive portion of the figure appears animated. 

20 38. The method of Claim 37, wherein the pattern 
data forming a succession of textures comprises 
pattern data from selected portions of a plurality 
of texture maps. 

25 39. The method of Claim 38, wherein the stored data 
representative of the surface includes data de- 
fining a plurality of points of the geometry of the 
surface, and wherein said step of associating the 
data representative of the surface to a texture 

30 coordinate space comprises the steps of storing 

with said surface data: 

first association data which references 
said surface to said texture coordinate space; 
and 

35 second association data which associ- 

ates different ones of said plurality of points of 
the geometry of the surface to coordinates of 
said texture coordinate space. 

40 40. The method of CI aim 39, wherein said first asso- 
ciation data references a physical location in the 
texture memory. 

41. The method of Claim 40. wherein said physical 
45 location in the texture memory is referenced by 

an indexed pointer. 

42. The method of Claim 37, wherein said step of se- 
quentially associating the stored pattern data for 

so different ones of said succession of textures to 

said texture coordinate space comprises se- 
quentially transferring said stored pattern data 
for different ones of said succession of textures 
to a designated physical area in the texture mem- 

55 ory corresponding to said texture coordinate 

space. 

43. A method for interchanging textures applied to a 
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surface situated in a three-dimensional virtual 
world generated by a computer graphic system 
to render a figure in action, the method compris- 
ing the steps of. 

defining a geometry for the surface; 

aligning said geometry with coordinates 
of a texture coordinate space in a texture mem- 
ory; 

generating pattern data from a succes- 
sion of at least three temporally related source 
images depicting the figure in action at different 
moments in time in the form of a succession of 
at least three time-sequenced textures, each 
pattern residing in a corresponding texture coor- 
dinate system; 

sequentially transferring the pattern data 
for different ones of said succession of time- 
sequenced textures to said texture coordinate 
space, said coordinate systems of said sequen- 
tially transferred texture pattern having a com- 
mon orientation in said texture coordinate space; 

storing three-dimensional data represen- 
tative of a plurality of surfaces forming the virtual 
world; and 

rendering in real time images of the virtual 
world based on the three-dimensional data rep- 
resentative of the plurality of surfaces, and a se- 
quence of images based on the geometry of the 
surface and the sequentially transferred texture 
patterns, to display the figure in action in the vir- 
tual world on at least a portion of the surface as 
texture which changes at intervals over a period 
of time so that the figure appears animated. 

44. A met hod for use by a texture mapping computer 
graphic system for rendering the image of a sur- 
face situated in a computer-generated three- 
dimensional virtual world, the method compris- 
ing the steps of: 

defining a geometry for the surface; 

aligning said geometry with coordinates 
of a texture coordinate space in a texture mem- 
ory; 

generating pattern data representative of 
a texture, said pattern data residing in a texture 
coordinate system; 

transferring the pattern data representa- 
tive of the texture to said texture coordinate 
space; 

storing three-dimensional data represen- 
tative of a plurality of surfaces forming the virtual 
world; 

rendering a first image of the virtual world 
based on the three-dimensional data represen- 
tative of the plurality of surfaces, including an im- 
age of the surface based on the defined geom- 
etry of the surface and the texture pattern data 
in said texture coordinate space; 



detecting an interaction between the 
viewer and the virtual world; 

in response to the detected interaction, 
modifying the pattern data representative of the 
5 texture in said texture coordinate space; and 

rendering a second image of the virtual 
world based on the three-dimensional data rep- 
resentative of the plurality of surfaces, including 
an image of the surface based on the defined 
10 geometry of the surface and the modified tex- 

ture pattern data in said texture coordinate 
space. 

45. A method for animating a figure in action in a 
15 three-dimensional virtual world rendered by a 

computer graphic system with time-sequenced 
images derived from a recording of a succession 
of at least three temporally related real world 
source images depicting the figure in action at 
20 different moments in time, the rendition capable 

of being displayed from a plurality of viewpoints 
looking into the virtual world from specific posi- 
tions and orientations, the method comprising 
the steps of: 

25 converting a plurality of selected images 

from said recording into data representative of a 
succession of at least three time-sequenced tex- 
tures; 

storing three-dimensional data represen- 
30 tative of a selected surface and a plurality of 

other surfaces forming the three-dimensional 
virtual world; and 

rendering in real time images of the virtual 
world based on three-dimensional data repre- 
ss sen tative of the plurality of other surfaces, in- 
cluding a sequence of images based on data rep- 
resentative of the selected surface and data rep- 
resentative of subsequent ones of said succes- 
sion of textures, to display the figure in action in 
40 the virtual world on at least a portion of the se- 
lected surface as texture which changes at inter- 
vals over a period of time so that the figure ap- 
pears animated. 

45 46. A method for displaying an animated subject in 
action at a position in a three-dimensional virtual 
world generated by a three-dimensional comput- 
er graphics sytem, the method comprising the 
steps of: 

so storing data representative of at least one 

surface; 

storing a succession of at least three time- 
sequenced textures representative of a corre- 
sponding succession of at least three temporally 
55 related source images depicting the subject in 

action at different moments in time; 

storing three-dimensional data represen- 
tative of a plurality of other surfaces forming the 
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virtual world; 

rendering in real time images of the virtual 
world based on the three-dimensional data rep- 
resentative of the plurality of other surfaces, and 
a sequence of images of the subject based on 
the data representative of the at least one sur- 
face and the succession of time- sequenced tex- 
tures; wherein: 

the sequence of images of the subject is 
rendered by mapping the plurality of time- 
sequenced textures onto the at least one surface 
to display the subject in action at the position in 
the virtual world as texture which changes at in- 
tervals over a period of time so that the subject 
appears animated. 

47. A met hod for use by a computer graphics system 
for displaying an animated likeness of a subject 
in action at a specific three-dimensional position 
in a computer-generated three-dimensional vir- 
tual world, the rendition capable of being dis- 
played from viewpoints having specific positions 
and orientations within the virtual world, the 
method comprising the steps of: 

storing three-dimensional data represen- 
tative of at least one surface positioned in the vir- 
tual world; 

storing data representative of a succes- 
sion of at least three time-sequenced textures 
derived from a recorded succession of at least 
three temporally related source images depicting 
the subject in action; and 

rendering in real time a sequence of im- 
ages of the subject relative to different ones of 
the viewpoints by mapping the succession of 
textures onto the at least one surface to display 
the subject in action as texture which changes at 
intervals over a period of time sothatthe subject 
appears animated. 

48. The method of Claim 46 or 47, wherein the suc- 
cession of textures are mapped onto at least a 
portion of the at least one surface. 

49. The method of Claim 46 or 47, wherein the step 
of mapping the succession of textures onto the 
at least one surface comprises the steps of as- 
signing a texture coordinate space to said at 
least one surface and sequentially associating 
the succession of textures with the texture coor- 
dinate space. 

50. The method of Claim 49, wherein the step of as- 
signing the texture coordinate space to the at 
least one surface comprises the step of assign- 
ing an area of texture memory to the at least one 
surface, and wherein the step of sequentially as- 
sociating the succession of textures with the tex- 



ture coordinate space comprises the step of se- 
quentially transferring the succession of textures 
into the area of texture memory. 

5 51. The method of Claim 46 or 47, wherein the step 
of mapping the succession of textures onto the 
at least one surface comprises the steps of as- 
sociating the succession of textures with a plur- 
ality of texture coordinate spaces and sequen- 

10 tially assigning the plurality of texture coordinate 

spaces to the at least one surface. 

52. The method of Claim 51 , wherein the step of as- 
sociating the succession of textures with the 

15 plurality of texture coordinate spaces comprises 

the step of transferring the succession of tex- 
tures into a plurality of areas of texture memory, 
and wherein the step of sequentially assigning 
the plurality of texture coordinate spaces to the 

20 at least one surface comprises the step of se- 

quentially assigning the plurality of areas of tex- 
ture memory to the at least one surface. 

53. The method of Claim 46 or 47, wherein the step 
25 of mapping the succession of textures onto the 

at least one surface comprises the steps of: 

assigning a texture coordinate space to 
the at least one surface; 

transferring the succession of textures 
30 into different coordinate areas within the texture 

coordinate space; and 

sequentially defining a plurality of differ- 
ent relationships between coordinates of the tex- 
ture coordinate space and the at least one sur- 
as face to sequentially associate the succession of 
textures with the at least one surface. 

54. The method of Claim 46 or 47, wherein the step 
of storing data representative of at least one sur- 

40 face comprises the step of storing data represen- 

tative of a plurality of surfaces, and wherein the 
step of mapping the succession of textures onto 
the at least one surface comprises the steps of 
sequentially mapping different ones of the suc- 

45 cession of textures onto different ones of the 

plurality of surfaces. 

55. The method of Claim 54, wherein the step of 
storing data representative of a plurality of sur- 

so faces comprises the step of storing data repre- 

sentative of a plurality of geometries associated 
with an object, each geometry including data 
representative of one of the plurality of surfaces, 
and wherein the step of sequentially mapping 

55 different ones of the succession of textures onto 

different ones of the plurality of surfaces com- 
prises the steps of assigning different ones of 
the succession of textures to different ones of 

21 
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the plurality of surfaces and sequentially render- 
ing the plurality of geometries. 

56. The method of Claim 54, wherein the step of 
storing data representative of a plurality of sur- 5 
faces comprises the step of storing data repre- 
sentative of a plurality of objects each having an 
associated geometry, each geometry including 
data representative of one of the plurality of sur- 
faces, and wherein the step of sequentially map- 10 
ping different ones of the succession of textures 
onto different ones of the plurality of surfaces 
comprises the steps of assigning different ones 

of the succession of textures to different ones of 

the plurality of surfaces and sequentially render- 1 5 

in g the plurality of objects. 

57. An apparatus for animating a rendition of a figure 
in action at a position in a three-dimensional vir- 
tual world generated .by a three-dimensional 20 
computer graphic system, the apparatus com- 
prising: 

first storage means for storing data repre- 
sentative of a first surface and three- 
dimensional data representative of a plurality of 25 
other surfaces forming the virtual world; 

second storage means for storing data de- 
rived from a succession of at least three tempor- 
ally related source images depicting the figure in 
action at different moments in time in the form of 30 
a succession of at least three time-sequenced 
textures; and 

means coupled to said first and second 
storage means for rendering in real time images 
of the virtual world based on three-dimensional 35 
data representative of the plurality of other sur- 
faces, and a sequence of images based on data 
representative of the first surface and data rep- 
resentative of subsequent ones of said succes- 
sion of textures, to display the figure in action in 40 
the virtual world on at least a portion of the sur- 
face as texture which changes at intervals over 
a period of time so that the figure appears ani- 
mated. 

45 

58. The apparatus of Claim 57, wherein the data rep- 
resentative of the first surface are three- 
dimensional data. 

59. The apparatus of Claim 57, wherein: so 

said first storage means stores data rep- 
resentative of a second surface; 

said second storage means stores data 
derived from a succession of at least three tem- 
porally related source images depicting a second 55 
figure in action at different moments in time in 
the form of a second succession of at least three 
time-sequenced textures; and 



said rendering means renders in real time 
a sequence of images based on the data repre- 
sentative of the second surface and the data 
representative of subsequent ones of said sec- 
ond succession of textures to display the second 
figure in action on at least a portion of the second 
surface as texture which changes at intervals 
over a period of time so that the second figure 
appears animated, 

whereby two texture-animated figures in 
action are displayed in the virtual world. 

60. An apparatus for animating a rendition of a figure 
in action at a position in a three-dimensional vir- 
tual world generated by a three-dimensional 
computer graphic system, the apparatus com- 
prising: 

first storage means for storing data repre- 
sentative of a first surface and three- 
dimensional data representative of a plurality of 
other surfaces forming the virtual world; 

second storage means for storing data de- 
rived from a succession of at least three tempor- 
ally related source images depicting the figure in 
action at different moments in time in the form of 
a succession of at least three time-sequenced 
textures; 

a texture memory; 

means for associating the first surface to 
said texture memory; 

means coupled to said second storage 
means and to said texture memory for sequen- 
tially transferring to said texture memory the 
stored texture data for subsequent ones of said 
succession of time-sequenced textures; and 

means coupled to said first storage 
means and said texture memory for rendering in 
real time images of the virtual world based on 
the three-dimensional data representative of the 
plurality of other surfaces, and a sequence of im- 
ages based on the stored data representative of 
the first surface and the sequentially transferred 
texture data in said texture memory, to display 
the figure in action in the virtual world on at least 
a portion of the first surface as texture which 
changes at intervals over a period of time so that 
the figure appears animated. 

61. The apparatus of Claim 60, wherein said asso- 
ciating means associates the first surface with a 
predetermined location in said texture memory. 

62. The apparatus of Claim 60, wherein: 

said first storage means stores data rep- 
resentative of a second surface; 

said second storage means stores data 
derived from a succession of at least three tem- 
porally related source images depicting a second 
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figure in action at different moments in time in 
the form of a second succession of at least three 
time- sequenced textures; 

said associating means associates the 
second surface to said texture memory; 5 

said sequential transfer means sequen- 
tially transfers to said texture memory the stored 
texture data for subsequent ones of said second 
succession of time-sequenced textures; and 

said rendering means renders in real time 10 
a sequence of images based on stored data rep- 
resentative of the second surface and sequen- 
tially transferred second texture data in said tex- 
ture memory to display the second figure in ac- 
tion on at least a portion of the second surface 15 
as texture which changes at intervals over a per- 
iod of lime so that the second figure appears ani- 
mated. 

whereby two texture-animated figures in 
action are displayed in the virtual world. 20 

63. The apparatus of Claim 57 or 62, wherein the two 
texture-animated figures in action are displayed 
simultaneously. 

25 

64. The apparatus of Claim 63, when dependent on 
Claim 57. wherein the data representative of the 
first and second surfaces are three-dimensional 
data. 

30 

65. The apparatus of Claim 57 or 60, wherein: 

the succession of time-sequenced tex- 
tures derives from a recorded succession of at 
least three temporally related source images of 
an entity existing in the real world; and 3t 

the texture displayed appears to be an 
animated likeness of the entity. 



66. The apparatus of Claim 57 or 60, wherein: 

the succession of time-sequenced tex- 40 
tures derives from a succession of at least three 
temporally related drawings. 

67. The apparatus of Claim 65 or 66, wherein the 
drawings comprise cartoon drawings and the 45 
texture displayed appears to be an animated car- 
toon. 

68. An apparatus for animating a rendition of a figure 

in action on an object situated in a three- so 
dimensional virtual world generated by a com- 
puter graphic system, the object including a plur- 
ality of surfaces, and the rendition capable of be- 
ing displayed from a plurality of viewpoints look- 
ing into the virtual world from specific positions 55 
and orientations, the apparatus comprising: 

first storage means for storing three- 
dimensional data representative of the surfaces 



of the object; 

second storage means for storing data de- 
rived from a succession of at least three tempor- 
ally related source images depicting the figure in 
action at different moments in time in the form of 
a succession of at least three time-sequenced 
textures; and 

means coupled to the first and second 
storage means for rendering in real time a se- 
quence of images based on the three- 
dimensional data representative of the surfaces 
and the data representative of subsequent ones 
of said succession of textures to display the ob- 
ject relative to different ones of said plurality of 
viewpoints with texture applied to at least a por- 
tion of at least one of the surfaces with the tex- 
ture changing at intervals over a period of time so 
that the figure appears animated. 

69. The apparatus of Claim 68, wherein said render- 
ing means causes texture to be applied over the 
plurality of surfaces with the texture over those 
surfaces changing so as to appear animated. 

70. The apparatus of Claim 69, wherein: 

the succession of time-sequenced tex- 
tures derives from a recorded succession of at 
least three temporally related source images of 
at least a portion of an entity existing in the real 
world; and 

the plurality of surfaces model an object 
having a three-dimensional shape approximat- 
ing the three-dimensional shape of at least that 
portion of the entity, 

whereby different perspective views of 
the texture can be displayed for different ones of 
the plurality of viewpoints such that the object 
displayed appears to be an animated three- 
dimensional likeness of at least that portion of 
the entity. 

71. In a computer graphics system, an apparatus for 
rendering a likeness of a real-world living being 
at a specif ic three-dimensional position in a com- 
puter-generated three-dimensional virtual 
world, the rendition capable of being displayed 
from viewpoints having specific positions and 
orientations within the virtual world, the appara- 
tus comprising: 

first storage means for storing three- 
dimensional data representative of at least one 
surface positioned in the virtual world; 

second storage means for storing data 
representative of a succession of at least three 
time-sequenced textures derived from a record- 
ed succession of at least three temporally related 
source images depicting the being at different 
moments in time; and 
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means coupled to the first and second 
storage means for rendering in real time a se- 
quence of images based on the three- 
dimensional data representative of the at least 
one surface and the data representative of sub- 5 
sequent ones of said succession of textures to 
display the being relative to different ones of said 
viewpoints as texture applied to at least a portion 
of the surface in the virtual world, with the tex- 
ture changing at intervals over a period of time so 10 
that the being appears animated. 

72. The apparatus of Claim 71 , wherein: 

the three-dimensional data are represen- 
tative of a plurality of surfaces modeling an ob- 15 
ject positioned in the virtual world, the object 
having a three-dimensional shape approximate- 
ly a three-dimensional shape of the being; and 

the changing texture is applied over the 
plurality of surfaces, 20 

whereby different perspective views of 
the texture can be displayed for different ones of 
the plurality of viewpoints such that the object 
displayed appears to be animated three- 
dimensional likeness of the being. 25 

73. In a computer graphic system, an apparatus for 
rendering a likeness of a real-world living being 
in a computer-generated three-dimensional vir- 
tual world, the apparatus comprising: 30 

first storage means for storing data repre- 
sentative of a surface positioned in the virtual 
world; 

second storage means for storing data 
representative of a succession of at least three 35 
time-sequenced textures derived from a record- 
ed succession of at least three temporally related 
source images depicting the being at different 
moments in time; and 

means coupled to the first and second 40 
storage means for rendering in real time a se- 
quence of images based on the data represen- 
tative of the surface and the data representative 
of subsequent ones of said succession of tex- 
tures to display the being in the virtual world on 45 
at least a portion of the surface as texture which 
changes at intervals over a period of time so that 
the being appears animated. 

74. An apparatus for animating a figure in action on so 
an object against a background in a three- 
dimensional virtual world rendered by a comput- 
er graphic system, the object having at least one 
surface and the rendition capable of being dis- 
played from viewpoints having specific positions 55 
and orientations within the virtual world, the ap- 
paratus comprising: 

first storage means for storing data repre- 



sentative of the at least one surface; 

second storage means for storing data de- 
rived from a succession of at least three tempor- 
ally related source images depicting the figure in 
action in a scene at different moments in time in 
the form of a succession of at least three time- 
sequenced textures, said data including informa- 
tion identifying a portion of texture including at 
least a portion of the figure as texture desired to 
be visible in the display; 

means coupled to said first and second 
storage means for rendering in real time a se- 
quence of images based on the data represen- 
tative of the surface and the data representative 
of subsequent ones of said succession of tex- 
tures to display relative to different ones of said 
viewpoints, on at least a portion of the surface of 
the object in the virtual world, the texture desired 
to be visible at least obscuring the background 
and changing at intervals over a period of time so 
that the figure appears animated. 

75. The apparatus of Claim 74, wherein the texture 
desired to be visible occludes the background. 

76. The apparatus of Claim 74, wherein the portion 
of the time-sequenced textures desired to be 
visible comprises a succession of at least three 
temporally related likenesses of a being existing 
the real world. 

77. The apparatus of Claim 76, wherein the likeness- 
es derive from recorded images of the being. 

78. The apparatus of Claims 72, 73 or 76, wherein 
the being comprises an animal. 

79. In a computer graphic system, an apparatus for 
rendering the image of a real-world entity situat- 
ed against a background in a computer-generat- 
ed three-dimensional virtual world, the rendition 
capable of being displayed from viewpoints hav- 
ing specific positions and orientations within the 
virtual world, the apparatus comprising: 

first storage means for storing data repre- 
sentative of a surface; 

second storage means for storing data 
representative of a succession of at least three 
temporally related likenesses of the entity re- 
corded at different moments in time in front of a 
backdrop as a succession of at least three tex- 
tures, said data including information identifying 
the entity to be visible and the backdrop to be in- 
visible in the display; and 

means coupled to said first and second 
storage means for rendering in real time a se- 
quence of images based on the data represen- 
tative of the surface and the data representative 
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of subsequent ones of said succession of tex- 
tures to display relative to different ones of said 
viewpoints, as texture on the surface, the like- 
nesses of the entity in front of the background, 
but not the backdrop, with the displayed texture 
changing at intervals over a period of time so that 
the entity appears animated. 

80. The apparatus of Claims 65, 70, 71 , 73, 76 or 79, 
wherein the entity or being includes a person. 

81. The apparatus of Claim 80, wherein the person 
is an actor performing a scripted event. 

82. The apparatus of Claim 81, wherein the actor 
performs in front of a backdrop and the data 
forming the succession of time-sequenced tex- 
tures includes information identifying the back- 
drop to be not visible in the display. 

83. The apparatus of Claims 71 , 73, 76 or 79, where- 
in the being is a plant 

84. An apparatus for animating a figure in action on 
a moving object having at least-one surface in a 
three-dimensional virtual world rendered by a 
computer graphic system, the moving object be- 
ing animated with images derived from a record- 
ing of a succession of at least three temporally 
related source images depicting a real world en- 
tity translationally moving in front of a backdrop 
in a manner analog to a desired transtational 
movement of the object in the virtual world, the 
rendition capable of being displayed from a plur- 
ality of viewpoints looking into the virtual world 
from specific positions and orientations, the ap- 
paratus comprising: 

means for converting a plurality of select- 
ed images from said recording into data repre- 
sentative of a succession of at least three time- 
sequenced textures, the texture data including 
information identifying the entity to be visible in 
the display; 

first storage means for storing three- 
dimensional data representative of the at least 
one surface of the moving object and 

means coupled to said converting means 
and to said first storage means for rendering in 
real time a sequence of images based on the 
data representative of the surface and the data 
representative of subsequent ones of said suc- 
cession of textures to display relative to different 
ones of said plurality of viewpoints t he surface of 
the object textured with the likeness of the enti- 
ty, with the texture changing at intervals over a 
period of time so that the entity appears animat- 
ed in synchronism with the object's movement. 



85. The apparatus of Claim 84, wherein the record- 
ing is a video recording. 

86. The apparatus of Claim 84, wherein the record- 
5 ing is a film recording. 

87. The apparatus of Claim 84, wherein the record- 
ing comprises a set of still photographs. 

10 88. The apparatus of any of Claims 80 to 84, wherein 
the likeness(es) is a photographic likeness. 

89. The apparatus of Claim 84, wherein: 

the entity is a person; 
15 the selected images are of the person 

walking; and 

the moving object appears as the person 
walking in synchronism with the person's trans- 
lational movement. 

20 

90. In a computer graphic system, an apparatus for 
rendering a likeness of a real-world human being 
situated in a computer-generated three- 
dimensional virtual world, the rendition capable 

25 of being displayed from a plurality of viewpoints 

looking into the virtual world from specific posi- 
tions and orientations, the apparatus compris- 
ing: 

first storage means for storing three- 

30 dimensional data representative of surfaces 

forming the virtual world; 

second storage means for storing data 
representative of a succession of at least three 
time-sequenced textures derived from a record- 

35 ed succession of at least three temporally related 

source images depicting the human being; and 
means coupled to the first and second 
storage means for rendering in real time images 
of the virtual world based upon the three- 

40 dimensional surface data and a sequence of im- 

ages of the human being based upon the data 
representative of a subsequent ones of said suc- 
cession of textures, whereby the human being is 
displayed in the virtual world relative to different 

45 ones of said plurality of viewpoints at a specific 

three-dimensional position as texture which 
changes so that the image of the human being 
appears animated. 

so 91. The apparatus of Claim 90, wherein the human 
being is an actor performing a scripted event 

92. An apparatus for animating a figure in action in 
a three-dimensional virtual world rendered by a 
55 computer graphic system with time-sequenced 

images derived from a recording of a succession 
of at least three temporally related real world 
source images depicting the figure in action at 
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different moments in time, the rendition capable 
of being displayed from a plurality of viewpoints 
looking into the virtual world from specific posi- 
tions and orientations, the apparatus compris- 
ing: 5 

means for converting a plurality of select- 
ed images from said recording into data repre- 
sentative of a succession of at least three time- 
sequenced textures; 

first storage means for storing three- 10 
dimensional data representative of a selected 
surface and a plurality of other surfaces forming 
the three-dimensional virtual world; and 

means coupled to said converting means 
and to said first storage means for rendering in is 
real time images of the virtual world based on 
three-dimensional data representative of the 
plurality of other surfaces, including a sequence 
of images based on data representative of the 
selected surface and data representative of sub- 20 
sequent ones of said succession of textures, to 
display the figure in action relative to different 
ones of said plurality of viewpoints in the virtual 
world on at least a portion of the selected sur- 
face as texture which changes at intervals over 25 
a period of time so thai the figure appears ani- 
mated. 

93. An apparatus for interchanging textures applied 

to a surface situated in a three-dimensional vir- 30 
tual world generated by a computer graphic sys- 
tem to render a figure with a portion in action, the 
apparatus comprising: 

means for storing data representative of 
the surface; 35 

means for storing pattern data represen- 
tative of a figure in the form of a first texture; 

means for storing pattern data derived 
from a succession of at least three temporally re- 
lated source images depicting an active portion 40 
of the figure at different moments in time in the 
form a succession of at least three time- 
sequenced textures; 

means for associating the data represen- 
tative of the surface to a texture coordinate 45 
space in a texture memory; 

means for associating the stored pattern 
data for the first texture to said texture coordin- 
ate space; 

means for sequentially associating the so 
stored pattern data for different ones of said suc- 
cession of time-sequenced textures to an area of 
said texture coordinate space within said first 
texture; 

means for storing three-dimensional data 55 
representative of a plurality of surfaces forming 
the virtual world; and 

means for rendering in real time images of 



the virtual world based on the three-dimensional 
data representative of the plurality of surfaces, 
and a sequence of images based on the stored 
data representative of the surface and the first 
and sequentially associated texture patterns, to 
display the figure in the virtual world on at least 
a portion of the surface as texture which 
changes in part at intervals over a period of time 
so that the active portion of the figure appears 
animated. 

94. The apparatus of Claim 93, wherein the pattern 
data forming a succession of textures comprises 
pattern data from selected portions of a plurality 
of texture maps. 

95. The apparatus of Claim 94, wherein the stored 
data representative of the surface includes data 
defining a plurality of points of the geometry of 
the surface, and wherein said means for asso- 
ciating the data representative of the surface to 
a texture coordinate space includes means for 
storing with said surface data: 

first association data which references 
said surface to said texture coordinate space; 
and 

second association data which associ- 
ates different ones of said plurality of points of 
the geometry of the surface to coordinates of 
said texture coordinate space. 

96. The apparatus of Claim 95, wherein said first as- 
sociation data references a physical location in 
the texture memory. 

97. The apparatus of Claim 96, wherein said physical 
location in the texture memory is referenced by 
an indexed pointer. 

98. The apparatus of Claim 93, wherein said means 
for sequentially associating the stored pattern 
data for different ones of said succession of tex- 
tures to said texture coordinate space comprises 
means for sequentially transferring said stored 
pattern data for different ones of said succession 
of textures to a designated physical area in the 
texture memory corresponding tc said texture 
coordinate space. 

99. An apparatus for interchanging textures applied 
to a surface situated in a three-dimensional vir- 
tual world generated by a computer graphic sys- 
tem to render a figure in action, the apparatus 
comprising: 

means for defining a geometry for the 
surface; 

means for aligning said geometry with co- 
ordinates of a texture coordinate space in a tex- 
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ture memory; 

means for generating pattern data from a 
succession of at least three temporally related 
source images depicting the figure in action at 
different moments in time in the form of a sue- s 
cession of at least three time-sequenced tex- 
tures, each pattern residing in a corresponding 
texture coordinate system; 

means for sequentially transferring the 
pattern data for different ones of said succession 10 
of time-sequenced textures to said texture coor- 
dinate space, said coordinate systems of said se- 
quentially transferring texture patterns having a 
common orientation in said texture coordinate 
space; * 5 

means for storing three-dimensional data 
representative of a plurality of surfaces forming 
the virtual world; and 

means for rendering in real time images of 
the virtual world based on the three-dimensional 20 
data representative of the plurality of surfaces, 
and a sequence of images based on the geom- 
etry of the surface and the sequentially transfer- 
red texture patterns, to display the figure in ac- 
tion in the virtual world on at least a portion of the 25 
surface as texture which changes at intervals 
over time so that the figure appears animated. 

1 00. An apparatus for use by a texture mapping com- 
puter graphic system for rendering the image of 30 
a surface situated in a computer-generated 
three-dimensional virtual world, the apparatus 
comprising: 

means for defining a geometry for the 
surface; 35 

means for aligning said geometry with co- 
ordinates of a texture coordinate space in a tex- 
ture memory; 

means for generating pattern data repre- 
sentative of a texture, said pattern-data residing 40 
in a texture coordinate system; 

means for transferring the pattern data 
representative of the texture to said texture co- 
ordinate space; 

means for storing three-dimensional data 45 
representative of a plurality of surfaces forming 
the virtual world; 

means for rendering a first image of the 
virtual world based on the three-dimensional 
data representative of the plurality of surfaces, so 
including an image of the surface based on the 
defined geometry of the surface and the texture 
pattern data in said texture coordinate space; 

means for detecting an interaction be- 
tween the viewer and the virtual world; 55 

means for modifying the pattern data rep- 
resentative of the texture in said texture coordin- 
ate space in response to a detected interaction; 



and 

means for rendering a second image of 
the virtual world based on the three-dimensional 
data representative of the plurality of surfaces, 
including an image of the surface based on the 
defined geometry of the surface and the modi- 
fied texture pattern data in said texture coordin- 
ate space. 

101. An apparatus for animating a figure in action in 
a three-dimensional virtual world rendered by a 
computer graphic system with time-sequenced 
images derived from a recording of a succession 
of at least three temporally related real world 
source images depicting the figure in action at 
different moments in time, the rendition capable 
of being displayed from a plurality of viewpoints 
looking into the virtual world from specific posi- 
tions and orientations, the apparatus compris- 
ing: 

means for converting a plurality of select- 
ed images from said recording into data repre- 
sentative of a succession of at least three time- 
sequenced textures; 

means for storing three-dimensional data 
representative of a selected surface and a plur- 
ality of other surfaces forming the three- 
dimensional virtual world; and 

means for rendering in real time images of 
the virtual world based on three-dimensional 
data representative of the plurality of other sur- 
faces, including a sequence of images based on 
data representative of the selected surface and 
data representative of subsequent ones of said 
succession of textures, to display the figure in 
action relative to different ones of the plurality of 
viewpoints in the virtual world on at least a por- 
tion of the selected surface as texture which 
changes so that the figure appears animated. 

102. An apparatus for displaying an animated subject 
in action at a position in a three-dimensional vir- 
tual world generated by a three-dimensional 
computer graphic system, the apparatus com- 
prising: 

means for storing data representative of at 
least one surface; 

means for storing a succession of at least 
three time-sequenced textures representative of 
a corresponding succession of at least three 
temporally related source images depicting the 
subject in action at different moments in time; 

means for storing three-dimensional data 
representative of a plurality of other surfaces 
forming the virtual world; 

means for rendering in real time images of 
the virtual world based on the three-dimensional 
data representative of the plurality of other sur- 
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faces, and a sequence of images of the subject 
based on the data representative of the at least 
one surface and the succession of time- 
sequence d textures; wherein: 

the sequence of image of the subject is 
rendered by mapping the succession of time- 
sequenced textures onto the at least one surface 
to display the subject in action at the position in 
the virtual world as texture which changes at in- 
tervals over a period of time so that the subject 
appears animated. 

1 03. An apparatus for use by a computer graphic sys- 
tem for displaying an animated likeness of a sub- 
ject in action at a specif ic three-dimensional pos- 
ition in a computer-generated three-dimensional 
virtual world, the rendition capable of being dis- 
played from viewpoints having specific positions 
and orientations within the virtual world, the ap- 
paratus comprising: 

means for storing three-dimensional data 
representative of at least one surface positioned 
in the virtual world; 

means for storing data representative of a 
succession of at least three time-sequenced tex- 
tures derived from a recorded succession of at 
least three temporally related source images de- 
picting the subject in action; and 

means for rendering in real time a se- 
quence of images of the subject relative to differ- 
ent ones of the viewpoints by mapping the suc- 
cession of textures onto the at least one surface 
to display the subject in action as texture which 
changes at intervals over a period of time so that 
the subject appears animated. 

104. The apparatus of Claim 102 or 103, wherein the 
means for mapping the succession of textures 
onto the at least one surface comprises means 
for assigning a texture coordinate space to said 40 
at least one surface and for sequentially asso- 
ciating the succession of textures with the tex- 
ture coordinate space. 

105. The apparatus of Claim 104, wherein the means 45 
for assigning the texture coordinate space to the 

at least one surface comprises means for assign- 
ing an area of texture memory to the at least one 
surface, and wherein the means for sequentially 
associating the succession of textures with the so 
texture coordinate space comprises means for 
sequentially transferring the plurality of textures 
into the area of texture memory. 

1 06. The apparatus of Claim 1 02 or 1 03, wherein the 55 
means for mapping the succession of textures 
onto the at least one surface comprises means 

for associating the succession of textures with a 



plurality of texture coordinate spaces and for se- 
quentially assigning the plurality of texture coor- 
dinate spaces to the at least one surface. 



110. The apparatus of Claim 109, wherein the means 
for storing data representative of a plurality of 
surfaces comprises means for storing data rep- 
resentative of a plurality of geometries associat- 
ed with an object, each geometry including data 
representative of one of the plurality of surfaces, 
and wherein the means for sequentially mapping 
different ones of the succession of textures onto 
different ones of the plurality of surfaces com- 
prises means for assigning different ones of the 
succession of textures to different ones of the 
plurality of surfaces and for sequentially render- 
ing the plurality of geometries. 

111. The apparatus of Claim 1 09, wherein the means 
for storing data representative of a plurality of 
surfaces comprises means for storing data rep- 
resentative of a plurality of objects each having 
an associated geometry, each geometry includ- 



5 107. The apparatus of Claim 106, wherein the means 
for associating the succession of textures with 
the plurality of texture coordinate spaces com- 
prises means for transferring the succession of 
textures into a plurality of areas of texture mem- 
10 ory, and wherein the means for sequentially as- 

signing the plurality of texture coordinate spaces 
to the at least one surface comprises means for 
sequentially assigning the plurality of areas of 
texture memory to the at least one surface. 

75 

108. The apparatus of Claim 102 or 103, wherein the 
means for mapping the succession of textures 
onto the at least one surface comprises: 

means for assigning a texture coordinate 
20 space to the at least one surface; 

means for transferring the succession of 
textures into different coordinate areas within 
the texture coordinate space; and 

means for sequentially defining a plurality 
25 of different relationship between coordinates of 

the texture coordinate space and the at I east one 
surface to sequentially associate the plurality of 
succession with the at least one surface. 

30 109. The apparatus of Claim 102 or 103, wherein the 
means for storing data representative of at least 
one of surface comprises means for storing data 
representative of a plurality of surfaces, and 
wherein the means for mapping the succession 
35 of textures onto the at least one surface compris- 

es means for sequentially mapping different 
ones of the succession of textures onto different 
ones of the plurality of surfaces. 
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ing data representative of one of the plurality of 
surfaces, and wherein the means for sequential- 
ly mapping different ones of the succession of 
textures onto different ones of the plurality of 
surfaces comprises means for assigning differ- s 
ent ones of the success ton of textures to differ- 
ent ones of the plurality of surfaces and for se- 
quentially rendering the plurality of objects. 

112. The apparatus of Claim 102 and 103, wherein 10 
the succession of time- sequenced textures are 
mapped onto at least a portion of the at least one 
surface. 
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